{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 0
   },
   "source": [
    "# 权重衰减\n",
    ":label:`sec_weight_decay`\n",
    "\n",
    "我们已经描述了过拟合的问题，现在我们可以介绍一些正则化模型的技术。我们总是可以通过去收集更多的训练数据来缓解过拟合。但这可能成本很高而且耗时.或者完全超出我们的控制，在短期内不可能做到。假设已经拥有尽可能多的高质量数据，现在我们将重点放在正则化技术上。\n",
    "\n",
    "回想一下，在多项式回归的例子（ :numref:`sec_model_selection`）中，我们可以通过调整拟合多项式的阶数来限制模型的容量。实际上，限制特征的数量是缓解过拟合的一种常用技术。然而，简单地丢弃特征对于这项工作来说可能过于生硬。我们继续思考多项式回归的例子，考虑高维输入可能发生的情况。多项式对多变量数据的自然扩展称为*单项式*（monomials），也可以说是变量幂的乘积。单项式的阶数是幂的和。例如，$x_1^2 x_2$和$x_3 x_5^2$都是3次单项式。\n",
    "\n",
    "注意，随着阶数$d$的增长，带有阶数$d$的项数迅速增加。给定$k$个变量，阶数$d$（即$k$多选$d$）的个数为${k - 1 + d} \\choose {k - 1}$。即使是阶数上的微小变化，比如从$2$到$3$，也会显著增加我们模型的复杂性。因此，我们经常需要一个更细粒度的工具来调整函数的复杂性。\n",
    "\n",
    "## 范数与权重衰减\n",
    "\n",
    "在之前的章节，我们已经描述了$L_2$范数和$L_1$范数，它们是$L_p$范数的特殊情况。\n",
    "(~~权重衰减是最广泛使用的正则化的技术之一~~)\n",
    "在训练参数化机器学习模型时，*权重衰减*（通常称为$L_2$正则化）是最广泛使用的正则化的技术之一。这项技术是基于一个基本直觉，即在所有函数$f$中，函数$f = 0$（所有输入都得到值$0$）在某种意义上是最简单的，我们可以通过函数与零的距离来衡量函数的复杂度。但是我们应该如何精确地测量一个函数和零之间的距离呢？没有一个正确的答案。事实上，整个数学分支，包括函数分析和巴拿赫空间理论，都在致力于回答这个问题。\n",
    "\n",
    "一种简单的方法是通过线性函数$f(\\mathbf{x}) = \\mathbf{w}^\\top \\mathbf{x}$中的权重向量的某个范数来度量其复杂性，例如$\\| \\mathbf{w} \\|^2$。要保证权重向量比较小，最常用方法是将其范数作为惩罚项加到最小化损失的问题中。将原来的训练目标*最小化训练标签上的预测损失*，调整为*最小化预测损失和惩罚项之和*。\n",
    "现在，如果我们的权重向量增长的太大，我们的学习算法可能会更集中于最小化权重范数$\\| \\mathbf{w} \\|^2$。这正是我们想要的。让我们回顾一下 :numref:`sec_linear_regression` 中的线性回归例子。我们的损失由下式给出：\n",
    "\n",
    "$$L(\\mathbf{w}, b) = \\frac{1}{n}\\sum_{i=1}^n \\frac{1}{2}\\left(\\mathbf{w}^\\top \\mathbf{x}^{(i)} + b - y^{(i)}\\right)^2.$$\n",
    "\n",
    "回想一下，$\\mathbf{x}^{(i)}$是样本$i$的特征，$y^{(i)}$是样本$i$的标签。$(\\mathbf{w}, b)$是权重和偏置参数。为了惩罚权重向量的大小，我们必须以某种方式在损失函数中添加$\\| \\mathbf{w} \\|^2$，但是模型应该如何平衡这个新的额外惩罚的损失？实际上，我们通过*正则化常数*$\\lambda$来描述这种权衡，这是一个非负超参数，我们使用验证数据拟合：\n",
    "\n",
    "$$L(\\mathbf{w}, b) + \\frac{\\lambda}{2} \\|\\mathbf{w}\\|^2,$$\n",
    "\n",
    "对于$\\lambda = 0$，我们恢复了原来的损失函数。对于$\\lambda > 0$，我们限制$\\| \\mathbf{w} \\|$的大小。我们仍然除以$2$：当我们取一个二次函数的导数时，$2$和$1/2$会抵消，以确保更新表达式看起来既漂亮又简单。聪明的读者可能会想知道为什么我们使用平方范数而不是标准范数（即欧几里得距离）。我们这样做是为了便于计算。通过平方$L_2$范数，我们去掉平方根，留下权重向量每个分量的平方和。这使得惩罚的导数很容易计算：导数的和等于和的导数。\n",
    "\n",
    "此外，你可能会问为什么我们首先使用$L_2$范数，而不是$L_1$范数。事实上，这些选择在整个统计领域中都是有效的和受欢迎的。$L_2$正则化线性模型构成经典的*岭回归*（ridge regression）算法，$L_1$正则化线性回归是统计学中类似的基本模型，通常被称为*套索回归*（lasso regression）。\n",
    "\n",
    "使用$L_2$范数的一个原因是它对权重向量的大分量施加了巨大的惩罚。这使得我们的学习算法偏向于在大量特征上均匀分布权重的模型。在实践中，这可能使它们对单个变量中的观测误差更为鲁棒。相比之下，$L_1$惩罚会导致模型将其他权重清除为零而将权重集中在一小部分特征上。这称为*特征选择*（feature selection），这可能是其他场景下需要的。\n",
    "\n",
    "使用与 :eqref:`eq_linreg_batch_update` 中的相同符号，$L_2$正则化回归的小批量随机梯度下降更新如下式：\n",
    "\n",
    "$$\n",
    "\\begin{aligned}\n",
    "\\mathbf{w} & \\leftarrow \\left(1- \\eta\\lambda \\right) \\mathbf{w} - \\frac{\\eta}{|\\mathcal{B}|} \\sum_{i \\in \\mathcal{B}} \\mathbf{x}^{(i)} \\left(\\mathbf{w}^\\top \\mathbf{x}^{(i)} + b - y^{(i)}\\right).\n",
    "\\end{aligned}\n",
    "$$\n",
    "\n",
    "根据之前章节所讲的，我们根据估计值与观测值之间的差异来更新$\\mathbf{w}$。然而，我们同时也在试图将$\\mathbf{w}$的大小缩小到零。这就是为什么这种方法有时被称为*权重衰减*。我们仅考虑惩罚项，优化算法在训练的每一步*衰减*权重。与特征选择相比，权重衰减为我们提供了一种连续的机制来调整函数的复杂度。较小的$\\lambda$值对应较少约束的$\\mathbf{w}$，而较大的$\\lambda$值对$\\mathbf{w}$的约束更大。\n",
    "\n",
    "是否对相应的偏置$b^2$进行惩罚在不同的实现中会有所不同。在神经网络的不同层中也会有所不同。通常，我们不正则化网络输出层的偏置项。\n",
    "\n",
    "## 高维线性回归\n",
    "\n",
    "我们通过一个简单的例子来说明演示权重衰减。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "origin_pos": 2,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import paddle\n",
    "from paddle import nn\n",
    "from d2l import torch as d2l\n",
    "import pd2l"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 4
   },
   "source": [
    "首先，我们[**像以前一样生成一些数据**]，生成公式如下：\n",
    "\n",
    "(**$$y = 0.05 + \\sum_{i = 1}^d 0.01 x_i + \\epsilon \\text{ where }\n",
    "\\epsilon \\sim \\mathcal{N}(0, 0.01^2).$$**)\n",
    "\n",
    "我们选择标签是关于输入的线性函数。标签同时被均值为0，标准差为0.01高斯噪声破坏。为了使过拟合的效果更加明显，我们可以将问题的维数增加到$d = 200$，并使用一个只包含20个样本的小训练集。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\WeiWu-GU\\anaconda3\\envs\\pte\\lib\\site-packages\\ipykernel\\ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
      "  and should_run_async(code)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'2.1.2'"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "paddle.__version__\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "origin_pos": 5,
    "scrolled": true,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [],
   "source": [
    "n_train, n_test, num_inputs, batch_size = 20, 100, 200, 5\n",
    "true_w, true_b = paddle.ones((num_inputs, 1)) * 0.01, 0.05\n",
    "train_data = pd2l.synthetic_data(true_w, true_b, n_train)\n",
    "train_iter = pd2l.load_array(train_data, batch_size)\n",
    "test_data = pd2l.synthetic_data(true_w, true_b, n_test)\n",
    "test_iter = pd2l.load_array(test_data, batch_size, is_train=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tuple"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(train_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(Tensor(shape=[20, 200], dtype=float32, place=CPUPlace, stop_gradient=True,\n",
       "        [[ 0.86401010,  2.32242107,  0.34629369, ...,  0.64903355,  0.83809000, -1.04373300],\n",
       "         [-0.73521167,  0.05137059, -1.77054071, ..., -0.28919482,  0.77603829, -0.50875086],\n",
       "         [ 1.92321503, -1.75834250, -0.68442386, ..., -0.24101926,  0.23858142, -1.45901525],\n",
       "         ...,\n",
       "         [ 1.53099728, -1.48775959, -1.21892989, ..., -0.04684093,  0.10560162,  0.76853377],\n",
       "         [-0.91647738,  1.15996015,  2.48777223, ...,  0.20768452, -0.77063823, -0.41458350],\n",
       "         [ 1.12426829,  0.69061953,  0.74275875, ..., -0.24119864, -0.10088623,  0.09340183]]),\n",
       " Tensor(shape=[20, 1], dtype=float32, place=CPUPlace, stop_gradient=True,\n",
       "        [[-0.03428355],\n",
       "         [ 0.06536283],\n",
       "         [ 0.00113643],\n",
       "         [-0.06883086],\n",
       "         [-0.05243002],\n",
       "         [ 0.26020166],\n",
       "         [ 0.45077348],\n",
       "         [ 0.07656152],\n",
       "         [ 0.16389121],\n",
       "         [ 0.00243414],\n",
       "         [-0.06865289],\n",
       "         [ 0.02741427],\n",
       "         [ 0.06025096],\n",
       "         [-0.09458974],\n",
       "         [ 0.06134989],\n",
       "         [ 0.05277457],\n",
       "         [ 0.05579818],\n",
       "         [-0.01781737],\n",
       "         [-0.13945857],\n",
       "         [-0.02532475]]))"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 6
   },
   "source": [
    "## 从零开始实现\n",
    "\n",
    "在下面，我们将从头开始实现权重衰减，只需将$L_2$的平方惩罚添加到原始目标函数中。\n",
    "\n",
    "### [**初始化模型参数**]\n",
    "\n",
    "首先，我们将定义一个函数来随机初始化我们的模型参数。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "origin_pos": 8,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [],
   "source": [
    "def init_params():\n",
    "    w = paddle.normal(0, 1, shape=(num_inputs, 1))\n",
    "    w.stop_gradient = False\n",
    "    b = paddle.zeros(shape=[1])\n",
    "    b.stop_gradient = False\n",
    "    return [w, b]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 10
   },
   "source": [
    "### (**定义$L_2$范数惩罚**)\n",
    "\n",
    "实现这一惩罚最方便的方法是对所有项求平方后并将它们求和。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "origin_pos": 12,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [],
   "source": [
    "def l2_penalty(w):\n",
    "    return paddle.sum(w.pow(2)) / 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 14
   },
   "source": [
    "### [**定义训练代码实现**]\n",
    "\n",
    "下面的代码将模型拟合训练数据集，并在测试数据集上进行评估。从 :numref:`chap_linear` 以来，线性网络和平方损失没有变化，所以我们通过`d2l.linreg`和`d2l.squared_loss`导入它们。唯一的变化是损失现在包括了惩罚项。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "origin_pos": 16,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [],
   "source": [
    "def train(lambd):\n",
    "    w, b = init_params()\n",
    "    net, loss = lambda X: pd2l.linreg(X, w, b), pd2l.squared_loss\n",
    "    num_epochs, lr = 100, 0.003\n",
    "    animator = pd2l.Animator(xlabel='epochs', ylabel='loss', yscale='log',\n",
    "                            xlim=[5, num_epochs], legend=['train', 'test'])\n",
    "    for epoch in range(num_epochs):\n",
    "        for X, y in train_iter():\n",
    "            # 增加了L2范数惩罚项，广播机制使l2_penalty(w)成为一个长度为`batch_size`的向量。\n",
    "            l = loss(net(X), y) + lambd * l2_penalty(w)\n",
    "            l.sum().backward()\n",
    "            pd2l.sgd([w, b], lr, batch_size)\n",
    "        if (epoch + 1) % 5 == 0:\n",
    "            animator.add(epoch + 1, (d2l.evaluate_loss(net, train_iter, loss),\n",
    "                                     d2l.evaluate_loss(net, test_iter, loss)))\n",
    "    print('w的L2范数是：', paddle.norm(w).item())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 18
   },
   "source": [
    "### [**忽略正则化直接训练**]\n",
    "\n",
    "我们现在用`lambd = 0`禁用权重衰减后运行这个代码。注意，这里训练误差有了减少，但测试误差没有减少。这意味着出现了严重的过拟合。这是过拟合的一个典型例子。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "origin_pos": 19,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "w的L2范数是： 12.657686233520508\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\r\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\r\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\r\n",
       "<svg height=\"180.65625pt\" version=\"1.1\" viewBox=\"0 0 263.421875 180.65625\" width=\"263.421875pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\r\n",
       " <metadata>\r\n",
       "  <rdf:RDF xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\r\n",
       "   <cc:Work>\r\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\r\n",
       "    <dc:date>2021-09-12T16:49:59.522565</dc:date>\r\n",
       "    <dc:format>image/svg+xml</dc:format>\r\n",
       "    <dc:creator>\r\n",
       "     <cc:Agent>\r\n",
       "      <dc:title>Matplotlib v3.4.3, https://matplotlib.org/</dc:title>\r\n",
       "     </cc:Agent>\r\n",
       "    </dc:creator>\r\n",
       "   </cc:Work>\r\n",
       "  </rdf:RDF>\r\n",
       " </metadata>\r\n",
       " <defs>\r\n",
       "  <style type=\"text/css\">*{stroke-linecap:butt;stroke-linejoin:round;}</style>\r\n",
       " </defs>\r\n",
       " <g id=\"figure_1\">\r\n",
       "  <g id=\"patch_1\">\r\n",
       "   <path d=\"M 0 180.65625 \r\n",
       "L 263.421875 180.65625 \r\n",
       "L 263.421875 0 \r\n",
       "L 0 0 \r\n",
       "z\r\n",
       "\" style=\"fill:none;\"/>\r\n",
       "  </g>\r\n",
       "  <g id=\"axes_1\">\r\n",
       "   <g id=\"patch_2\">\r\n",
       "    <path d=\"M 51.378125 143.1 \r\n",
       "L 246.678125 143.1 \r\n",
       "L 246.678125 7.2 \r\n",
       "L 51.378125 7.2 \r\n",
       "z\r\n",
       "\" style=\"fill:#ffffff;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"matplotlib.axis_1\">\r\n",
       "    <g id=\"xtick_1\">\r\n",
       "     <g id=\"line2d_1\">\r\n",
       "      <path clip-path=\"url(#pbec66d021b)\" d=\"M 82.214967 143.1 \r\n",
       "L 82.214967 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_2\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 0 0 \r\n",
       "L 0 3.5 \r\n",
       "\" id=\"mbaff4a64ca\" style=\"stroke:#000000;stroke-width:0.8;\"/>\r\n",
       "      </defs>\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"82.214967\" xlink:href=\"#mbaff4a64ca\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_1\">\r\n",
       "      <!-- 20 -->\r\n",
       "      <g transform=\"translate(75.852467 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 1228 531 \r\n",
       "L 3431 531 \r\n",
       "L 3431 0 \r\n",
       "L 469 0 \r\n",
       "L 469 531 \r\n",
       "Q 828 903 1448 1529 \r\n",
       "Q 2069 2156 2228 2338 \r\n",
       "Q 2531 2678 2651 2914 \r\n",
       "Q 2772 3150 2772 3378 \r\n",
       "Q 2772 3750 2511 3984 \r\n",
       "Q 2250 4219 1831 4219 \r\n",
       "Q 1534 4219 1204 4116 \r\n",
       "Q 875 4013 500 3803 \r\n",
       "L 500 4441 \r\n",
       "Q 881 4594 1212 4672 \r\n",
       "Q 1544 4750 1819 4750 \r\n",
       "Q 2544 4750 2975 4387 \r\n",
       "Q 3406 4025 3406 3419 \r\n",
       "Q 3406 3131 3298 2873 \r\n",
       "Q 3191 2616 2906 2266 \r\n",
       "Q 2828 2175 2409 1742 \r\n",
       "Q 1991 1309 1228 531 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-32\" transform=\"scale(0.015625)\"/>\r\n",
       "        <path d=\"M 2034 4250 \r\n",
       "Q 1547 4250 1301 3770 \r\n",
       "Q 1056 3291 1056 2328 \r\n",
       "Q 1056 1369 1301 889 \r\n",
       "Q 1547 409 2034 409 \r\n",
       "Q 2525 409 2770 889 \r\n",
       "Q 3016 1369 3016 2328 \r\n",
       "Q 3016 3291 2770 3770 \r\n",
       "Q 2525 4250 2034 4250 \r\n",
       "z\r\n",
       "M 2034 4750 \r\n",
       "Q 2819 4750 3233 4129 \r\n",
       "Q 3647 3509 3647 2328 \r\n",
       "Q 3647 1150 3233 529 \r\n",
       "Q 2819 -91 2034 -91 \r\n",
       "Q 1250 -91 836 529 \r\n",
       "Q 422 1150 422 2328 \r\n",
       "Q 422 3509 836 4129 \r\n",
       "Q 1250 4750 2034 4750 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-30\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_2\">\r\n",
       "     <g id=\"line2d_3\">\r\n",
       "      <path clip-path=\"url(#pbec66d021b)\" d=\"M 123.330757 143.1 \r\n",
       "L 123.330757 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_4\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"123.330757\" xlink:href=\"#mbaff4a64ca\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_2\">\r\n",
       "      <!-- 40 -->\r\n",
       "      <g transform=\"translate(116.968257 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 2419 4116 \r\n",
       "L 825 1625 \r\n",
       "L 2419 1625 \r\n",
       "L 2419 4116 \r\n",
       "z\r\n",
       "M 2253 4666 \r\n",
       "L 3047 4666 \r\n",
       "L 3047 1625 \r\n",
       "L 3713 1625 \r\n",
       "L 3713 1100 \r\n",
       "L 3047 1100 \r\n",
       "L 3047 0 \r\n",
       "L 2419 0 \r\n",
       "L 2419 1100 \r\n",
       "L 313 1100 \r\n",
       "L 313 1709 \r\n",
       "L 2253 4666 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-34\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-34\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_3\">\r\n",
       "     <g id=\"line2d_5\">\r\n",
       "      <path clip-path=\"url(#pbec66d021b)\" d=\"M 164.446546 143.1 \r\n",
       "L 164.446546 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_6\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"164.446546\" xlink:href=\"#mbaff4a64ca\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_3\">\r\n",
       "      <!-- 60 -->\r\n",
       "      <g transform=\"translate(158.084046 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 2113 2584 \r\n",
       "Q 1688 2584 1439 2293 \r\n",
       "Q 1191 2003 1191 1497 \r\n",
       "Q 1191 994 1439 701 \r\n",
       "Q 1688 409 2113 409 \r\n",
       "Q 2538 409 2786 701 \r\n",
       "Q 3034 994 3034 1497 \r\n",
       "Q 3034 2003 2786 2293 \r\n",
       "Q 2538 2584 2113 2584 \r\n",
       "z\r\n",
       "M 3366 4563 \r\n",
       "L 3366 3988 \r\n",
       "Q 3128 4100 2886 4159 \r\n",
       "Q 2644 4219 2406 4219 \r\n",
       "Q 1781 4219 1451 3797 \r\n",
       "Q 1122 3375 1075 2522 \r\n",
       "Q 1259 2794 1537 2939 \r\n",
       "Q 1816 3084 2150 3084 \r\n",
       "Q 2853 3084 3261 2657 \r\n",
       "Q 3669 2231 3669 1497 \r\n",
       "Q 3669 778 3244 343 \r\n",
       "Q 2819 -91 2113 -91 \r\n",
       "Q 1303 -91 875 529 \r\n",
       "Q 447 1150 447 2328 \r\n",
       "Q 447 3434 972 4092 \r\n",
       "Q 1497 4750 2381 4750 \r\n",
       "Q 2619 4750 2861 4703 \r\n",
       "Q 3103 4656 3366 4563 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-36\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-36\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_4\">\r\n",
       "     <g id=\"line2d_7\">\r\n",
       "      <path clip-path=\"url(#pbec66d021b)\" d=\"M 205.562336 143.1 \r\n",
       "L 205.562336 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_8\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"205.562336\" xlink:href=\"#mbaff4a64ca\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_4\">\r\n",
       "      <!-- 80 -->\r\n",
       "      <g transform=\"translate(199.199836 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 2034 2216 \r\n",
       "Q 1584 2216 1326 1975 \r\n",
       "Q 1069 1734 1069 1313 \r\n",
       "Q 1069 891 1326 650 \r\n",
       "Q 1584 409 2034 409 \r\n",
       "Q 2484 409 2743 651 \r\n",
       "Q 3003 894 3003 1313 \r\n",
       "Q 3003 1734 2745 1975 \r\n",
       "Q 2488 2216 2034 2216 \r\n",
       "z\r\n",
       "M 1403 2484 \r\n",
       "Q 997 2584 770 2862 \r\n",
       "Q 544 3141 544 3541 \r\n",
       "Q 544 4100 942 4425 \r\n",
       "Q 1341 4750 2034 4750 \r\n",
       "Q 2731 4750 3128 4425 \r\n",
       "Q 3525 4100 3525 3541 \r\n",
       "Q 3525 3141 3298 2862 \r\n",
       "Q 3072 2584 2669 2484 \r\n",
       "Q 3125 2378 3379 2068 \r\n",
       "Q 3634 1759 3634 1313 \r\n",
       "Q 3634 634 3220 271 \r\n",
       "Q 2806 -91 2034 -91 \r\n",
       "Q 1263 -91 848 271 \r\n",
       "Q 434 634 434 1313 \r\n",
       "Q 434 1759 690 2068 \r\n",
       "Q 947 2378 1403 2484 \r\n",
       "z\r\n",
       "M 1172 3481 \r\n",
       "Q 1172 3119 1398 2916 \r\n",
       "Q 1625 2713 2034 2713 \r\n",
       "Q 2441 2713 2670 2916 \r\n",
       "Q 2900 3119 2900 3481 \r\n",
       "Q 2900 3844 2670 4047 \r\n",
       "Q 2441 4250 2034 4250 \r\n",
       "Q 1625 4250 1398 4047 \r\n",
       "Q 1172 3844 1172 3481 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-38\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-38\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_5\">\r\n",
       "     <g id=\"line2d_9\">\r\n",
       "      <path clip-path=\"url(#pbec66d021b)\" d=\"M 246.678125 143.1 \r\n",
       "L 246.678125 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_10\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"246.678125\" xlink:href=\"#mbaff4a64ca\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_5\">\r\n",
       "      <!-- 100 -->\r\n",
       "      <g transform=\"translate(237.134375 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 794 531 \r\n",
       "L 1825 531 \r\n",
       "L 1825 4091 \r\n",
       "L 703 3866 \r\n",
       "L 703 4441 \r\n",
       "L 1819 4666 \r\n",
       "L 2450 4666 \r\n",
       "L 2450 531 \r\n",
       "L 3481 531 \r\n",
       "L 3481 0 \r\n",
       "L 794 0 \r\n",
       "L 794 531 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-31\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"text_6\">\r\n",
       "     <!-- epochs -->\r\n",
       "     <g transform=\"translate(131.195312 171.376563)scale(0.1 -0.1)\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 3597 1894 \r\n",
       "L 3597 1613 \r\n",
       "L 953 1613 \r\n",
       "Q 991 1019 1311 708 \r\n",
       "Q 1631 397 2203 397 \r\n",
       "Q 2534 397 2845 478 \r\n",
       "Q 3156 559 3463 722 \r\n",
       "L 3463 178 \r\n",
       "Q 3153 47 2828 -22 \r\n",
       "Q 2503 -91 2169 -91 \r\n",
       "Q 1331 -91 842 396 \r\n",
       "Q 353 884 353 1716 \r\n",
       "Q 353 2575 817 3079 \r\n",
       "Q 1281 3584 2069 3584 \r\n",
       "Q 2775 3584 3186 3129 \r\n",
       "Q 3597 2675 3597 1894 \r\n",
       "z\r\n",
       "M 3022 2063 \r\n",
       "Q 3016 2534 2758 2815 \r\n",
       "Q 2500 3097 2075 3097 \r\n",
       "Q 1594 3097 1305 2825 \r\n",
       "Q 1016 2553 972 2059 \r\n",
       "L 3022 2063 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-65\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 1159 525 \r\n",
       "L 1159 -1331 \r\n",
       "L 581 -1331 \r\n",
       "L 581 3500 \r\n",
       "L 1159 3500 \r\n",
       "L 1159 2969 \r\n",
       "Q 1341 3281 1617 3432 \r\n",
       "Q 1894 3584 2278 3584 \r\n",
       "Q 2916 3584 3314 3078 \r\n",
       "Q 3713 2572 3713 1747 \r\n",
       "Q 3713 922 3314 415 \r\n",
       "Q 2916 -91 2278 -91 \r\n",
       "Q 1894 -91 1617 61 \r\n",
       "Q 1341 213 1159 525 \r\n",
       "z\r\n",
       "M 3116 1747 \r\n",
       "Q 3116 2381 2855 2742 \r\n",
       "Q 2594 3103 2138 3103 \r\n",
       "Q 1681 3103 1420 2742 \r\n",
       "Q 1159 2381 1159 1747 \r\n",
       "Q 1159 1113 1420 752 \r\n",
       "Q 1681 391 2138 391 \r\n",
       "Q 2594 391 2855 752 \r\n",
       "Q 3116 1113 3116 1747 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-70\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 1959 3097 \r\n",
       "Q 1497 3097 1228 2736 \r\n",
       "Q 959 2375 959 1747 \r\n",
       "Q 959 1119 1226 758 \r\n",
       "Q 1494 397 1959 397 \r\n",
       "Q 2419 397 2687 759 \r\n",
       "Q 2956 1122 2956 1747 \r\n",
       "Q 2956 2369 2687 2733 \r\n",
       "Q 2419 3097 1959 3097 \r\n",
       "z\r\n",
       "M 1959 3584 \r\n",
       "Q 2709 3584 3137 3096 \r\n",
       "Q 3566 2609 3566 1747 \r\n",
       "Q 3566 888 3137 398 \r\n",
       "Q 2709 -91 1959 -91 \r\n",
       "Q 1206 -91 779 398 \r\n",
       "Q 353 888 353 1747 \r\n",
       "Q 353 2609 779 3096 \r\n",
       "Q 1206 3584 1959 3584 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-6f\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 3122 3366 \r\n",
       "L 3122 2828 \r\n",
       "Q 2878 2963 2633 3030 \r\n",
       "Q 2388 3097 2138 3097 \r\n",
       "Q 1578 3097 1268 2742 \r\n",
       "Q 959 2388 959 1747 \r\n",
       "Q 959 1106 1268 751 \r\n",
       "Q 1578 397 2138 397 \r\n",
       "Q 2388 397 2633 464 \r\n",
       "Q 2878 531 3122 666 \r\n",
       "L 3122 134 \r\n",
       "Q 2881 22 2623 -34 \r\n",
       "Q 2366 -91 2075 -91 \r\n",
       "Q 1284 -91 818 406 \r\n",
       "Q 353 903 353 1747 \r\n",
       "Q 353 2603 823 3093 \r\n",
       "Q 1294 3584 2113 3584 \r\n",
       "Q 2378 3584 2631 3529 \r\n",
       "Q 2884 3475 3122 3366 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-63\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 3513 2113 \r\n",
       "L 3513 0 \r\n",
       "L 2938 0 \r\n",
       "L 2938 2094 \r\n",
       "Q 2938 2591 2744 2837 \r\n",
       "Q 2550 3084 2163 3084 \r\n",
       "Q 1697 3084 1428 2787 \r\n",
       "Q 1159 2491 1159 1978 \r\n",
       "L 1159 0 \r\n",
       "L 581 0 \r\n",
       "L 581 4863 \r\n",
       "L 1159 4863 \r\n",
       "L 1159 2956 \r\n",
       "Q 1366 3272 1645 3428 \r\n",
       "Q 1925 3584 2291 3584 \r\n",
       "Q 2894 3584 3203 3211 \r\n",
       "Q 3513 2838 3513 2113 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-68\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 2834 3397 \r\n",
       "L 2834 2853 \r\n",
       "Q 2591 2978 2328 3040 \r\n",
       "Q 2066 3103 1784 3103 \r\n",
       "Q 1356 3103 1142 2972 \r\n",
       "Q 928 2841 928 2578 \r\n",
       "Q 928 2378 1081 2264 \r\n",
       "Q 1234 2150 1697 2047 \r\n",
       "L 1894 2003 \r\n",
       "Q 2506 1872 2764 1633 \r\n",
       "Q 3022 1394 3022 966 \r\n",
       "Q 3022 478 2636 193 \r\n",
       "Q 2250 -91 1575 -91 \r\n",
       "Q 1294 -91 989 -36 \r\n",
       "Q 684 19 347 128 \r\n",
       "L 347 722 \r\n",
       "Q 666 556 975 473 \r\n",
       "Q 1284 391 1588 391 \r\n",
       "Q 1994 391 2212 530 \r\n",
       "Q 2431 669 2431 922 \r\n",
       "Q 2431 1156 2273 1281 \r\n",
       "Q 2116 1406 1581 1522 \r\n",
       "L 1381 1569 \r\n",
       "Q 847 1681 609 1914 \r\n",
       "Q 372 2147 372 2553 \r\n",
       "Q 372 3047 722 3315 \r\n",
       "Q 1072 3584 1716 3584 \r\n",
       "Q 2034 3584 2315 3537 \r\n",
       "Q 2597 3491 2834 3397 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-73\" transform=\"scale(0.015625)\"/>\r\n",
       "      </defs>\r\n",
       "      <use xlink:href=\"#DejaVuSans-65\"/>\r\n",
       "      <use x=\"61.523438\" xlink:href=\"#DejaVuSans-70\"/>\r\n",
       "      <use x=\"125\" xlink:href=\"#DejaVuSans-6f\"/>\r\n",
       "      <use x=\"186.181641\" xlink:href=\"#DejaVuSans-63\"/>\r\n",
       "      <use x=\"241.162109\" xlink:href=\"#DejaVuSans-68\"/>\r\n",
       "      <use x=\"304.541016\" xlink:href=\"#DejaVuSans-73\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "   <g id=\"matplotlib.axis_2\">\r\n",
       "    <g id=\"ytick_1\">\r\n",
       "     <g id=\"line2d_11\">\r\n",
       "      <path clip-path=\"url(#pbec66d021b)\" d=\"M 51.378125 125.442357 \r\n",
       "L 246.678125 125.442357 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_12\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 0 0 \r\n",
       "L -3.5 0 \r\n",
       "\" id=\"m0247291129\" style=\"stroke:#000000;stroke-width:0.8;\"/>\r\n",
       "      </defs>\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"51.378125\" xlink:href=\"#m0247291129\" y=\"125.442357\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_7\">\r\n",
       "      <!-- $\\mathdefault{10^{-5}}$ -->\r\n",
       "      <g transform=\"translate(20.878125 129.241576)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 678 2272 \r\n",
       "L 4684 2272 \r\n",
       "L 4684 1741 \r\n",
       "L 678 1741 \r\n",
       "L 678 2272 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-2212\" transform=\"scale(0.015625)\"/>\r\n",
       "        <path d=\"M 691 4666 \r\n",
       "L 3169 4666 \r\n",
       "L 3169 4134 \r\n",
       "L 1269 4134 \r\n",
       "L 1269 2991 \r\n",
       "Q 1406 3038 1543 3061 \r\n",
       "Q 1681 3084 1819 3084 \r\n",
       "Q 2600 3084 3056 2656 \r\n",
       "Q 3513 2228 3513 1497 \r\n",
       "Q 3513 744 3044 326 \r\n",
       "Q 2575 -91 1722 -91 \r\n",
       "Q 1428 -91 1123 -41 \r\n",
       "Q 819 9 494 109 \r\n",
       "L 494 744 \r\n",
       "Q 775 591 1075 516 \r\n",
       "Q 1375 441 1709 441 \r\n",
       "Q 2250 441 2565 725 \r\n",
       "Q 2881 1009 2881 1497 \r\n",
       "Q 2881 1984 2565 2268 \r\n",
       "Q 2250 2553 1709 2553 \r\n",
       "Q 1456 2553 1204 2497 \r\n",
       "Q 953 2441 691 2322 \r\n",
       "L 691 4666 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-35\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use transform=\"translate(0 0.684375)\" xlink:href=\"#DejaVuSans-31\"/>\r\n",
       "       <use transform=\"translate(63.623047 0.684375)\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use transform=\"translate(128.203125 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-2212\"/>\r\n",
       "       <use transform=\"translate(186.855469 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-35\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_2\">\r\n",
       "     <g id=\"line2d_13\">\r\n",
       "      <path clip-path=\"url(#pbec66d021b)\" d=\"M 51.378125 93.163377 \r\n",
       "L 246.678125 93.163377 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_14\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"51.378125\" xlink:href=\"#m0247291129\" y=\"93.163377\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_8\">\r\n",
       "      <!-- $\\mathdefault{10^{-3}}$ -->\r\n",
       "      <g transform=\"translate(20.878125 96.962596)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 2597 2516 \r\n",
       "Q 3050 2419 3304 2112 \r\n",
       "Q 3559 1806 3559 1356 \r\n",
       "Q 3559 666 3084 287 \r\n",
       "Q 2609 -91 1734 -91 \r\n",
       "Q 1441 -91 1130 -33 \r\n",
       "Q 819 25 488 141 \r\n",
       "L 488 750 \r\n",
       "Q 750 597 1062 519 \r\n",
       "Q 1375 441 1716 441 \r\n",
       "Q 2309 441 2620 675 \r\n",
       "Q 2931 909 2931 1356 \r\n",
       "Q 2931 1769 2642 2001 \r\n",
       "Q 2353 2234 1838 2234 \r\n",
       "L 1294 2234 \r\n",
       "L 1294 2753 \r\n",
       "L 1863 2753 \r\n",
       "Q 2328 2753 2575 2939 \r\n",
       "Q 2822 3125 2822 3475 \r\n",
       "Q 2822 3834 2567 4026 \r\n",
       "Q 2313 4219 1838 4219 \r\n",
       "Q 1578 4219 1281 4162 \r\n",
       "Q 984 4106 628 3988 \r\n",
       "L 628 4550 \r\n",
       "Q 988 4650 1302 4700 \r\n",
       "Q 1616 4750 1894 4750 \r\n",
       "Q 2613 4750 3031 4423 \r\n",
       "Q 3450 4097 3450 3541 \r\n",
       "Q 3450 3153 3228 2886 \r\n",
       "Q 3006 2619 2597 2516 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-33\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use transform=\"translate(0 0.765625)\" xlink:href=\"#DejaVuSans-31\"/>\r\n",
       "       <use transform=\"translate(63.623047 0.765625)\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use transform=\"translate(128.203125 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-2212\"/>\r\n",
       "       <use transform=\"translate(186.855469 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-33\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_3\">\r\n",
       "     <g id=\"line2d_15\">\r\n",
       "      <path clip-path=\"url(#pbec66d021b)\" d=\"M 51.378125 60.884397 \r\n",
       "L 246.678125 60.884397 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_16\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"51.378125\" xlink:href=\"#m0247291129\" y=\"60.884397\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_9\">\r\n",
       "      <!-- $\\mathdefault{10^{-1}}$ -->\r\n",
       "      <g transform=\"translate(20.878125 64.683616)scale(0.1 -0.1)\">\r\n",
       "       <use transform=\"translate(0 0.684375)\" xlink:href=\"#DejaVuSans-31\"/>\r\n",
       "       <use transform=\"translate(63.623047 0.684375)\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use transform=\"translate(128.203125 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-2212\"/>\r\n",
       "       <use transform=\"translate(186.855469 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-31\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_4\">\r\n",
       "     <g id=\"line2d_17\">\r\n",
       "      <path clip-path=\"url(#pbec66d021b)\" d=\"M 51.378125 28.605417 \r\n",
       "L 246.678125 28.605417 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_18\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"51.378125\" xlink:href=\"#m0247291129\" y=\"28.605417\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_10\">\r\n",
       "      <!-- $\\mathdefault{10^{1}}$ -->\r\n",
       "      <g transform=\"translate(26.778125 32.404635)scale(0.1 -0.1)\">\r\n",
       "       <use transform=\"translate(0 0.684375)\" xlink:href=\"#DejaVuSans-31\"/>\r\n",
       "       <use transform=\"translate(63.623047 0.684375)\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use transform=\"translate(128.203125 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-31\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"text_11\">\r\n",
       "     <!-- loss -->\r\n",
       "     <g transform=\"translate(14.798437 84.807812)rotate(-90)scale(0.1 -0.1)\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 603 4863 \r\n",
       "L 1178 4863 \r\n",
       "L 1178 0 \r\n",
       "L 603 0 \r\n",
       "L 603 4863 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-6c\" transform=\"scale(0.015625)\"/>\r\n",
       "      </defs>\r\n",
       "      <use xlink:href=\"#DejaVuSans-6c\"/>\r\n",
       "      <use x=\"27.783203\" xlink:href=\"#DejaVuSans-6f\"/>\r\n",
       "      <use x=\"88.964844\" xlink:href=\"#DejaVuSans-73\"/>\r\n",
       "      <use x=\"141.064453\" xlink:href=\"#DejaVuSans-73\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "   <g id=\"line2d_19\">\r\n",
       "    <path clip-path=\"url(#pbec66d021b)\" d=\"M 51.378125 22.870872 \r\n",
       "L 61.657072 32.132865 \r\n",
       "L 71.93602 40.490996 \r\n",
       "L 82.214967 47.964581 \r\n",
       "L 92.493914 54.718223 \r\n",
       "L 102.772862 60.989508 \r\n",
       "L 113.051809 66.940476 \r\n",
       "L 123.330757 72.667419 \r\n",
       "L 133.609704 78.264261 \r\n",
       "L 143.888651 83.766708 \r\n",
       "L 154.167599 89.224611 \r\n",
       "L 164.446546 94.627564 \r\n",
       "L 174.725493 99.987436 \r\n",
       "L 185.004441 105.334013 \r\n",
       "L 195.283388 110.650391 \r\n",
       "L 205.562336 115.935573 \r\n",
       "L 215.841283 121.210237 \r\n",
       "L 226.12023 126.459118 \r\n",
       "L 236.399178 131.699268 \r\n",
       "L 246.678125 136.922727 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-linecap:square;stroke-width:1.5;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"line2d_20\">\r\n",
       "    <path clip-path=\"url(#pbec66d021b)\" d=\"M 51.378125 13.706461 \r\n",
       "L 61.657072 13.636646 \r\n",
       "L 71.93602 13.553582 \r\n",
       "L 82.214967 13.492568 \r\n",
       "L 92.493914 13.451656 \r\n",
       "L 102.772862 13.425067 \r\n",
       "L 113.051809 13.408015 \r\n",
       "L 123.330757 13.39711 \r\n",
       "L 133.609704 13.390122 \r\n",
       "L 143.888651 13.385582 \r\n",
       "L 154.167599 13.382635 \r\n",
       "L 164.446546 13.380721 \r\n",
       "L 174.725493 13.379493 \r\n",
       "L 185.004441 13.378685 \r\n",
       "L 195.283388 13.378157 \r\n",
       "L 205.562336 13.377809 \r\n",
       "L 215.841283 13.377587 \r\n",
       "L 226.12023 13.377437 \r\n",
       "L 236.399178 13.377338 \r\n",
       "L 246.678125 13.377273 \r\n",
       "\" style=\"fill:none;stroke:#bf00bf;stroke-dasharray:5.55,2.4;stroke-dashoffset:0;stroke-width:1.5;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_3\">\r\n",
       "    <path d=\"M 51.378125 143.1 \r\n",
       "L 51.378125 7.2 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_4\">\r\n",
       "    <path d=\"M 246.678125 143.1 \r\n",
       "L 246.678125 7.2 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_5\">\r\n",
       "    <path d=\"M 51.378125 143.1 \r\n",
       "L 246.678125 143.1 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_6\">\r\n",
       "    <path d=\"M 51.378125 7.2 \r\n",
       "L 246.678125 7.2 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"legend_1\">\r\n",
       "    <g id=\"patch_7\">\r\n",
       "     <path d=\"M 58.378125 138.1 \r\n",
       "L 113.653125 138.1 \r\n",
       "Q 115.653125 138.1 115.653125 136.1 \r\n",
       "L 115.653125 107.74375 \r\n",
       "Q 115.653125 105.74375 113.653125 105.74375 \r\n",
       "L 58.378125 105.74375 \r\n",
       "Q 56.378125 105.74375 56.378125 107.74375 \r\n",
       "L 56.378125 136.1 \r\n",
       "Q 56.378125 138.1 58.378125 138.1 \r\n",
       "z\r\n",
       "\" style=\"fill:#ffffff;opacity:0.8;stroke:#cccccc;stroke-linejoin:miter;\"/>\r\n",
       "    </g>\r\n",
       "    <g id=\"line2d_21\">\r\n",
       "     <path d=\"M 60.378125 113.842187 \r\n",
       "L 80.378125 113.842187 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-linecap:square;stroke-width:1.5;\"/>\r\n",
       "    </g>\r\n",
       "    <g id=\"line2d_22\"/>\r\n",
       "    <g id=\"text_12\">\r\n",
       "     <!-- train -->\r\n",
       "     <g transform=\"translate(88.378125 117.342187)scale(0.1 -0.1)\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 1172 4494 \r\n",
       "L 1172 3500 \r\n",
       "L 2356 3500 \r\n",
       "L 2356 3053 \r\n",
       "L 1172 3053 \r\n",
       "L 1172 1153 \r\n",
       "Q 1172 725 1289 603 \r\n",
       "Q 1406 481 1766 481 \r\n",
       "L 2356 481 \r\n",
       "L 2356 0 \r\n",
       "L 1766 0 \r\n",
       "Q 1100 0 847 248 \r\n",
       "Q 594 497 594 1153 \r\n",
       "L 594 3053 \r\n",
       "L 172 3053 \r\n",
       "L 172 3500 \r\n",
       "L 594 3500 \r\n",
       "L 594 4494 \r\n",
       "L 1172 4494 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-74\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 2631 2963 \r\n",
       "Q 2534 3019 2420 3045 \r\n",
       "Q 2306 3072 2169 3072 \r\n",
       "Q 1681 3072 1420 2755 \r\n",
       "Q 1159 2438 1159 1844 \r\n",
       "L 1159 0 \r\n",
       "L 581 0 \r\n",
       "L 581 3500 \r\n",
       "L 1159 3500 \r\n",
       "L 1159 2956 \r\n",
       "Q 1341 3275 1631 3429 \r\n",
       "Q 1922 3584 2338 3584 \r\n",
       "Q 2397 3584 2469 3576 \r\n",
       "Q 2541 3569 2628 3553 \r\n",
       "L 2631 2963 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-72\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 2194 1759 \r\n",
       "Q 1497 1759 1228 1600 \r\n",
       "Q 959 1441 959 1056 \r\n",
       "Q 959 750 1161 570 \r\n",
       "Q 1363 391 1709 391 \r\n",
       "Q 2188 391 2477 730 \r\n",
       "Q 2766 1069 2766 1631 \r\n",
       "L 2766 1759 \r\n",
       "L 2194 1759 \r\n",
       "z\r\n",
       "M 3341 1997 \r\n",
       "L 3341 0 \r\n",
       "L 2766 0 \r\n",
       "L 2766 531 \r\n",
       "Q 2569 213 2275 61 \r\n",
       "Q 1981 -91 1556 -91 \r\n",
       "Q 1019 -91 701 211 \r\n",
       "Q 384 513 384 1019 \r\n",
       "Q 384 1609 779 1909 \r\n",
       "Q 1175 2209 1959 2209 \r\n",
       "L 2766 2209 \r\n",
       "L 2766 2266 \r\n",
       "Q 2766 2663 2505 2880 \r\n",
       "Q 2244 3097 1772 3097 \r\n",
       "Q 1472 3097 1187 3025 \r\n",
       "Q 903 2953 641 2809 \r\n",
       "L 641 3341 \r\n",
       "Q 956 3463 1253 3523 \r\n",
       "Q 1550 3584 1831 3584 \r\n",
       "Q 2591 3584 2966 3190 \r\n",
       "Q 3341 2797 3341 1997 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-61\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 603 3500 \r\n",
       "L 1178 3500 \r\n",
       "L 1178 0 \r\n",
       "L 603 0 \r\n",
       "L 603 3500 \r\n",
       "z\r\n",
       "M 603 4863 \r\n",
       "L 1178 4863 \r\n",
       "L 1178 4134 \r\n",
       "L 603 4134 \r\n",
       "L 603 4863 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-69\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 3513 2113 \r\n",
       "L 3513 0 \r\n",
       "L 2938 0 \r\n",
       "L 2938 2094 \r\n",
       "Q 2938 2591 2744 2837 \r\n",
       "Q 2550 3084 2163 3084 \r\n",
       "Q 1697 3084 1428 2787 \r\n",
       "Q 1159 2491 1159 1978 \r\n",
       "L 1159 0 \r\n",
       "L 581 0 \r\n",
       "L 581 3500 \r\n",
       "L 1159 3500 \r\n",
       "L 1159 2956 \r\n",
       "Q 1366 3272 1645 3428 \r\n",
       "Q 1925 3584 2291 3584 \r\n",
       "Q 2894 3584 3203 3211 \r\n",
       "Q 3513 2838 3513 2113 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-6e\" transform=\"scale(0.015625)\"/>\r\n",
       "      </defs>\r\n",
       "      <use xlink:href=\"#DejaVuSans-74\"/>\r\n",
       "      <use x=\"39.208984\" xlink:href=\"#DejaVuSans-72\"/>\r\n",
       "      <use x=\"80.322266\" xlink:href=\"#DejaVuSans-61\"/>\r\n",
       "      <use x=\"141.601562\" xlink:href=\"#DejaVuSans-69\"/>\r\n",
       "      <use x=\"169.384766\" xlink:href=\"#DejaVuSans-6e\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"line2d_23\">\r\n",
       "     <path d=\"M 60.378125 128.520312 \r\n",
       "L 80.378125 128.520312 \r\n",
       "\" style=\"fill:none;stroke:#bf00bf;stroke-dasharray:5.55,2.4;stroke-dashoffset:0;stroke-width:1.5;\"/>\r\n",
       "    </g>\r\n",
       "    <g id=\"line2d_24\"/>\r\n",
       "    <g id=\"text_13\">\r\n",
       "     <!-- test -->\r\n",
       "     <g transform=\"translate(88.378125 132.020312)scale(0.1 -0.1)\">\r\n",
       "      <use xlink:href=\"#DejaVuSans-74\"/>\r\n",
       "      <use x=\"39.208984\" xlink:href=\"#DejaVuSans-65\"/>\r\n",
       "      <use x=\"100.732422\" xlink:href=\"#DejaVuSans-73\"/>\r\n",
       "      <use x=\"152.832031\" xlink:href=\"#DejaVuSans-74\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "  </g>\r\n",
       " </g>\r\n",
       " <defs>\r\n",
       "  <clipPath id=\"pbec66d021b\">\r\n",
       "   <rect height=\"135.9\" width=\"195.3\" x=\"51.378125\" y=\"7.2\"/>\r\n",
       "  </clipPath>\r\n",
       " </defs>\r\n",
       "</svg>\r\n"
      ],
      "text/plain": [
       "<Figure size 252x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "train(lambd=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 20
   },
   "source": [
    "### [**使用权重衰减**]\n",
    "\n",
    "下面，我们使用权重衰减来运行代码。注意，在这里训练误差增大，但测试误差减小。这正是我们期望从正则化中得到的效果。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "origin_pos": 21,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "w的L2范数是： 0.34710878133773804\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\r\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\r\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\r\n",
       "<svg height=\"183.947366pt\" version=\"1.1\" viewBox=\"0 0 263.421875 183.947366\" width=\"263.421875pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\r\n",
       " <metadata>\r\n",
       "  <rdf:RDF xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\r\n",
       "   <cc:Work>\r\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\r\n",
       "    <dc:date>2021-09-12T16:50:09.684055</dc:date>\r\n",
       "    <dc:format>image/svg+xml</dc:format>\r\n",
       "    <dc:creator>\r\n",
       "     <cc:Agent>\r\n",
       "      <dc:title>Matplotlib v3.4.3, https://matplotlib.org/</dc:title>\r\n",
       "     </cc:Agent>\r\n",
       "    </dc:creator>\r\n",
       "   </cc:Work>\r\n",
       "  </rdf:RDF>\r\n",
       " </metadata>\r\n",
       " <defs>\r\n",
       "  <style type=\"text/css\">*{stroke-linecap:butt;stroke-linejoin:round;}</style>\r\n",
       " </defs>\r\n",
       " <g id=\"figure_1\">\r\n",
       "  <g id=\"patch_1\">\r\n",
       "   <path d=\"M 0 183.947366 \r\n",
       "L 263.421875 183.947366 \r\n",
       "L 263.421875 -0 \r\n",
       "L 0 -0 \r\n",
       "z\r\n",
       "\" style=\"fill:none;\"/>\r\n",
       "  </g>\r\n",
       "  <g id=\"axes_1\">\r\n",
       "   <g id=\"patch_2\">\r\n",
       "    <path d=\"M 51.378125 146.391116 \r\n",
       "L 246.678125 146.391116 \r\n",
       "L 246.678125 10.491116 \r\n",
       "L 51.378125 10.491116 \r\n",
       "z\r\n",
       "\" style=\"fill:#ffffff;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"matplotlib.axis_1\">\r\n",
       "    <g id=\"xtick_1\">\r\n",
       "     <g id=\"line2d_1\">\r\n",
       "      <path clip-path=\"url(#p923a6f6e7e)\" d=\"M 82.214967 146.391116 \r\n",
       "L 82.214967 10.491116 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_2\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 0 0 \r\n",
       "L 0 3.5 \r\n",
       "\" id=\"m4051c6378a\" style=\"stroke:#000000;stroke-width:0.8;\"/>\r\n",
       "      </defs>\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"82.214967\" xlink:href=\"#m4051c6378a\" y=\"146.391116\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_1\">\r\n",
       "      <!-- 20 -->\r\n",
       "      <g transform=\"translate(75.852467 160.989553)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 1228 531 \r\n",
       "L 3431 531 \r\n",
       "L 3431 0 \r\n",
       "L 469 0 \r\n",
       "L 469 531 \r\n",
       "Q 828 903 1448 1529 \r\n",
       "Q 2069 2156 2228 2338 \r\n",
       "Q 2531 2678 2651 2914 \r\n",
       "Q 2772 3150 2772 3378 \r\n",
       "Q 2772 3750 2511 3984 \r\n",
       "Q 2250 4219 1831 4219 \r\n",
       "Q 1534 4219 1204 4116 \r\n",
       "Q 875 4013 500 3803 \r\n",
       "L 500 4441 \r\n",
       "Q 881 4594 1212 4672 \r\n",
       "Q 1544 4750 1819 4750 \r\n",
       "Q 2544 4750 2975 4387 \r\n",
       "Q 3406 4025 3406 3419 \r\n",
       "Q 3406 3131 3298 2873 \r\n",
       "Q 3191 2616 2906 2266 \r\n",
       "Q 2828 2175 2409 1742 \r\n",
       "Q 1991 1309 1228 531 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-32\" transform=\"scale(0.015625)\"/>\r\n",
       "        <path d=\"M 2034 4250 \r\n",
       "Q 1547 4250 1301 3770 \r\n",
       "Q 1056 3291 1056 2328 \r\n",
       "Q 1056 1369 1301 889 \r\n",
       "Q 1547 409 2034 409 \r\n",
       "Q 2525 409 2770 889 \r\n",
       "Q 3016 1369 3016 2328 \r\n",
       "Q 3016 3291 2770 3770 \r\n",
       "Q 2525 4250 2034 4250 \r\n",
       "z\r\n",
       "M 2034 4750 \r\n",
       "Q 2819 4750 3233 4129 \r\n",
       "Q 3647 3509 3647 2328 \r\n",
       "Q 3647 1150 3233 529 \r\n",
       "Q 2819 -91 2034 -91 \r\n",
       "Q 1250 -91 836 529 \r\n",
       "Q 422 1150 422 2328 \r\n",
       "Q 422 3509 836 4129 \r\n",
       "Q 1250 4750 2034 4750 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-30\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_2\">\r\n",
       "     <g id=\"line2d_3\">\r\n",
       "      <path clip-path=\"url(#p923a6f6e7e)\" d=\"M 123.330757 146.391116 \r\n",
       "L 123.330757 10.491116 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_4\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"123.330757\" xlink:href=\"#m4051c6378a\" y=\"146.391116\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_2\">\r\n",
       "      <!-- 40 -->\r\n",
       "      <g transform=\"translate(116.968257 160.989553)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 2419 4116 \r\n",
       "L 825 1625 \r\n",
       "L 2419 1625 \r\n",
       "L 2419 4116 \r\n",
       "z\r\n",
       "M 2253 4666 \r\n",
       "L 3047 4666 \r\n",
       "L 3047 1625 \r\n",
       "L 3713 1625 \r\n",
       "L 3713 1100 \r\n",
       "L 3047 1100 \r\n",
       "L 3047 0 \r\n",
       "L 2419 0 \r\n",
       "L 2419 1100 \r\n",
       "L 313 1100 \r\n",
       "L 313 1709 \r\n",
       "L 2253 4666 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-34\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-34\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_3\">\r\n",
       "     <g id=\"line2d_5\">\r\n",
       "      <path clip-path=\"url(#p923a6f6e7e)\" d=\"M 164.446546 146.391116 \r\n",
       "L 164.446546 10.491116 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_6\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"164.446546\" xlink:href=\"#m4051c6378a\" y=\"146.391116\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_3\">\r\n",
       "      <!-- 60 -->\r\n",
       "      <g transform=\"translate(158.084046 160.989553)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 2113 2584 \r\n",
       "Q 1688 2584 1439 2293 \r\n",
       "Q 1191 2003 1191 1497 \r\n",
       "Q 1191 994 1439 701 \r\n",
       "Q 1688 409 2113 409 \r\n",
       "Q 2538 409 2786 701 \r\n",
       "Q 3034 994 3034 1497 \r\n",
       "Q 3034 2003 2786 2293 \r\n",
       "Q 2538 2584 2113 2584 \r\n",
       "z\r\n",
       "M 3366 4563 \r\n",
       "L 3366 3988 \r\n",
       "Q 3128 4100 2886 4159 \r\n",
       "Q 2644 4219 2406 4219 \r\n",
       "Q 1781 4219 1451 3797 \r\n",
       "Q 1122 3375 1075 2522 \r\n",
       "Q 1259 2794 1537 2939 \r\n",
       "Q 1816 3084 2150 3084 \r\n",
       "Q 2853 3084 3261 2657 \r\n",
       "Q 3669 2231 3669 1497 \r\n",
       "Q 3669 778 3244 343 \r\n",
       "Q 2819 -91 2113 -91 \r\n",
       "Q 1303 -91 875 529 \r\n",
       "Q 447 1150 447 2328 \r\n",
       "Q 447 3434 972 4092 \r\n",
       "Q 1497 4750 2381 4750 \r\n",
       "Q 2619 4750 2861 4703 \r\n",
       "Q 3103 4656 3366 4563 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-36\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-36\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_4\">\r\n",
       "     <g id=\"line2d_7\">\r\n",
       "      <path clip-path=\"url(#p923a6f6e7e)\" d=\"M 205.562336 146.391116 \r\n",
       "L 205.562336 10.491116 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_8\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"205.562336\" xlink:href=\"#m4051c6378a\" y=\"146.391116\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_4\">\r\n",
       "      <!-- 80 -->\r\n",
       "      <g transform=\"translate(199.199836 160.989553)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 2034 2216 \r\n",
       "Q 1584 2216 1326 1975 \r\n",
       "Q 1069 1734 1069 1313 \r\n",
       "Q 1069 891 1326 650 \r\n",
       "Q 1584 409 2034 409 \r\n",
       "Q 2484 409 2743 651 \r\n",
       "Q 3003 894 3003 1313 \r\n",
       "Q 3003 1734 2745 1975 \r\n",
       "Q 2488 2216 2034 2216 \r\n",
       "z\r\n",
       "M 1403 2484 \r\n",
       "Q 997 2584 770 2862 \r\n",
       "Q 544 3141 544 3541 \r\n",
       "Q 544 4100 942 4425 \r\n",
       "Q 1341 4750 2034 4750 \r\n",
       "Q 2731 4750 3128 4425 \r\n",
       "Q 3525 4100 3525 3541 \r\n",
       "Q 3525 3141 3298 2862 \r\n",
       "Q 3072 2584 2669 2484 \r\n",
       "Q 3125 2378 3379 2068 \r\n",
       "Q 3634 1759 3634 1313 \r\n",
       "Q 3634 634 3220 271 \r\n",
       "Q 2806 -91 2034 -91 \r\n",
       "Q 1263 -91 848 271 \r\n",
       "Q 434 634 434 1313 \r\n",
       "Q 434 1759 690 2068 \r\n",
       "Q 947 2378 1403 2484 \r\n",
       "z\r\n",
       "M 1172 3481 \r\n",
       "Q 1172 3119 1398 2916 \r\n",
       "Q 1625 2713 2034 2713 \r\n",
       "Q 2441 2713 2670 2916 \r\n",
       "Q 2900 3119 2900 3481 \r\n",
       "Q 2900 3844 2670 4047 \r\n",
       "Q 2441 4250 2034 4250 \r\n",
       "Q 1625 4250 1398 4047 \r\n",
       "Q 1172 3844 1172 3481 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-38\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-38\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_5\">\r\n",
       "     <g id=\"line2d_9\">\r\n",
       "      <path clip-path=\"url(#p923a6f6e7e)\" d=\"M 246.678125 146.391116 \r\n",
       "L 246.678125 10.491116 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_10\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"246.678125\" xlink:href=\"#m4051c6378a\" y=\"146.391116\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_5\">\r\n",
       "      <!-- 100 -->\r\n",
       "      <g transform=\"translate(237.134375 160.989553)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 794 531 \r\n",
       "L 1825 531 \r\n",
       "L 1825 4091 \r\n",
       "L 703 3866 \r\n",
       "L 703 4441 \r\n",
       "L 1819 4666 \r\n",
       "L 2450 4666 \r\n",
       "L 2450 531 \r\n",
       "L 3481 531 \r\n",
       "L 3481 0 \r\n",
       "L 794 0 \r\n",
       "L 794 531 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-31\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"text_6\">\r\n",
       "     <!-- epochs -->\r\n",
       "     <g transform=\"translate(131.195312 174.667678)scale(0.1 -0.1)\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 3597 1894 \r\n",
       "L 3597 1613 \r\n",
       "L 953 1613 \r\n",
       "Q 991 1019 1311 708 \r\n",
       "Q 1631 397 2203 397 \r\n",
       "Q 2534 397 2845 478 \r\n",
       "Q 3156 559 3463 722 \r\n",
       "L 3463 178 \r\n",
       "Q 3153 47 2828 -22 \r\n",
       "Q 2503 -91 2169 -91 \r\n",
       "Q 1331 -91 842 396 \r\n",
       "Q 353 884 353 1716 \r\n",
       "Q 353 2575 817 3079 \r\n",
       "Q 1281 3584 2069 3584 \r\n",
       "Q 2775 3584 3186 3129 \r\n",
       "Q 3597 2675 3597 1894 \r\n",
       "z\r\n",
       "M 3022 2063 \r\n",
       "Q 3016 2534 2758 2815 \r\n",
       "Q 2500 3097 2075 3097 \r\n",
       "Q 1594 3097 1305 2825 \r\n",
       "Q 1016 2553 972 2059 \r\n",
       "L 3022 2063 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-65\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 1159 525 \r\n",
       "L 1159 -1331 \r\n",
       "L 581 -1331 \r\n",
       "L 581 3500 \r\n",
       "L 1159 3500 \r\n",
       "L 1159 2969 \r\n",
       "Q 1341 3281 1617 3432 \r\n",
       "Q 1894 3584 2278 3584 \r\n",
       "Q 2916 3584 3314 3078 \r\n",
       "Q 3713 2572 3713 1747 \r\n",
       "Q 3713 922 3314 415 \r\n",
       "Q 2916 -91 2278 -91 \r\n",
       "Q 1894 -91 1617 61 \r\n",
       "Q 1341 213 1159 525 \r\n",
       "z\r\n",
       "M 3116 1747 \r\n",
       "Q 3116 2381 2855 2742 \r\n",
       "Q 2594 3103 2138 3103 \r\n",
       "Q 1681 3103 1420 2742 \r\n",
       "Q 1159 2381 1159 1747 \r\n",
       "Q 1159 1113 1420 752 \r\n",
       "Q 1681 391 2138 391 \r\n",
       "Q 2594 391 2855 752 \r\n",
       "Q 3116 1113 3116 1747 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-70\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 1959 3097 \r\n",
       "Q 1497 3097 1228 2736 \r\n",
       "Q 959 2375 959 1747 \r\n",
       "Q 959 1119 1226 758 \r\n",
       "Q 1494 397 1959 397 \r\n",
       "Q 2419 397 2687 759 \r\n",
       "Q 2956 1122 2956 1747 \r\n",
       "Q 2956 2369 2687 2733 \r\n",
       "Q 2419 3097 1959 3097 \r\n",
       "z\r\n",
       "M 1959 3584 \r\n",
       "Q 2709 3584 3137 3096 \r\n",
       "Q 3566 2609 3566 1747 \r\n",
       "Q 3566 888 3137 398 \r\n",
       "Q 2709 -91 1959 -91 \r\n",
       "Q 1206 -91 779 398 \r\n",
       "Q 353 888 353 1747 \r\n",
       "Q 353 2609 779 3096 \r\n",
       "Q 1206 3584 1959 3584 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-6f\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 3122 3366 \r\n",
       "L 3122 2828 \r\n",
       "Q 2878 2963 2633 3030 \r\n",
       "Q 2388 3097 2138 3097 \r\n",
       "Q 1578 3097 1268 2742 \r\n",
       "Q 959 2388 959 1747 \r\n",
       "Q 959 1106 1268 751 \r\n",
       "Q 1578 397 2138 397 \r\n",
       "Q 2388 397 2633 464 \r\n",
       "Q 2878 531 3122 666 \r\n",
       "L 3122 134 \r\n",
       "Q 2881 22 2623 -34 \r\n",
       "Q 2366 -91 2075 -91 \r\n",
       "Q 1284 -91 818 406 \r\n",
       "Q 353 903 353 1747 \r\n",
       "Q 353 2603 823 3093 \r\n",
       "Q 1294 3584 2113 3584 \r\n",
       "Q 2378 3584 2631 3529 \r\n",
       "Q 2884 3475 3122 3366 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-63\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 3513 2113 \r\n",
       "L 3513 0 \r\n",
       "L 2938 0 \r\n",
       "L 2938 2094 \r\n",
       "Q 2938 2591 2744 2837 \r\n",
       "Q 2550 3084 2163 3084 \r\n",
       "Q 1697 3084 1428 2787 \r\n",
       "Q 1159 2491 1159 1978 \r\n",
       "L 1159 0 \r\n",
       "L 581 0 \r\n",
       "L 581 4863 \r\n",
       "L 1159 4863 \r\n",
       "L 1159 2956 \r\n",
       "Q 1366 3272 1645 3428 \r\n",
       "Q 1925 3584 2291 3584 \r\n",
       "Q 2894 3584 3203 3211 \r\n",
       "Q 3513 2838 3513 2113 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-68\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 2834 3397 \r\n",
       "L 2834 2853 \r\n",
       "Q 2591 2978 2328 3040 \r\n",
       "Q 2066 3103 1784 3103 \r\n",
       "Q 1356 3103 1142 2972 \r\n",
       "Q 928 2841 928 2578 \r\n",
       "Q 928 2378 1081 2264 \r\n",
       "Q 1234 2150 1697 2047 \r\n",
       "L 1894 2003 \r\n",
       "Q 2506 1872 2764 1633 \r\n",
       "Q 3022 1394 3022 966 \r\n",
       "Q 3022 478 2636 193 \r\n",
       "Q 2250 -91 1575 -91 \r\n",
       "Q 1294 -91 989 -36 \r\n",
       "Q 684 19 347 128 \r\n",
       "L 347 722 \r\n",
       "Q 666 556 975 473 \r\n",
       "Q 1284 391 1588 391 \r\n",
       "Q 1994 391 2212 530 \r\n",
       "Q 2431 669 2431 922 \r\n",
       "Q 2431 1156 2273 1281 \r\n",
       "Q 2116 1406 1581 1522 \r\n",
       "L 1381 1569 \r\n",
       "Q 847 1681 609 1914 \r\n",
       "Q 372 2147 372 2553 \r\n",
       "Q 372 3047 722 3315 \r\n",
       "Q 1072 3584 1716 3584 \r\n",
       "Q 2034 3584 2315 3537 \r\n",
       "Q 2597 3491 2834 3397 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-73\" transform=\"scale(0.015625)\"/>\r\n",
       "      </defs>\r\n",
       "      <use xlink:href=\"#DejaVuSans-65\"/>\r\n",
       "      <use x=\"61.523438\" xlink:href=\"#DejaVuSans-70\"/>\r\n",
       "      <use x=\"125\" xlink:href=\"#DejaVuSans-6f\"/>\r\n",
       "      <use x=\"186.181641\" xlink:href=\"#DejaVuSans-63\"/>\r\n",
       "      <use x=\"241.162109\" xlink:href=\"#DejaVuSans-68\"/>\r\n",
       "      <use x=\"304.541016\" xlink:href=\"#DejaVuSans-73\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "   <g id=\"matplotlib.axis_2\">\r\n",
       "    <g id=\"ytick_1\">\r\n",
       "     <g id=\"line2d_11\">\r\n",
       "      <path clip-path=\"url(#p923a6f6e7e)\" d=\"M 51.378125 119.567356 \r\n",
       "L 246.678125 119.567356 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_12\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 0 0 \r\n",
       "L -3.5 0 \r\n",
       "\" id=\"mb15c192f53\" style=\"stroke:#000000;stroke-width:0.8;\"/>\r\n",
       "      </defs>\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"51.378125\" xlink:href=\"#mb15c192f53\" y=\"119.567356\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_7\">\r\n",
       "      <!-- $\\mathdefault{10^{-2}}$ -->\r\n",
       "      <g transform=\"translate(20.878125 123.366575)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 678 2272 \r\n",
       "L 4684 2272 \r\n",
       "L 4684 1741 \r\n",
       "L 678 1741 \r\n",
       "L 678 2272 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-2212\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use transform=\"translate(0 0.765625)\" xlink:href=\"#DejaVuSans-31\"/>\r\n",
       "       <use transform=\"translate(63.623047 0.765625)\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use transform=\"translate(128.203125 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-2212\"/>\r\n",
       "       <use transform=\"translate(186.855469 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-32\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_2\">\r\n",
       "     <g id=\"line2d_13\">\r\n",
       "      <path clip-path=\"url(#p923a6f6e7e)\" d=\"M 51.378125 92.425322 \r\n",
       "L 246.678125 92.425322 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_14\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"51.378125\" xlink:href=\"#mb15c192f53\" y=\"92.425322\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_8\">\r\n",
       "      <!-- $\\mathdefault{10^{-1}}$ -->\r\n",
       "      <g transform=\"translate(20.878125 96.224541)scale(0.1 -0.1)\">\r\n",
       "       <use transform=\"translate(0 0.684375)\" xlink:href=\"#DejaVuSans-31\"/>\r\n",
       "       <use transform=\"translate(63.623047 0.684375)\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use transform=\"translate(128.203125 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-2212\"/>\r\n",
       "       <use transform=\"translate(186.855469 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-31\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_3\">\r\n",
       "     <g id=\"line2d_15\">\r\n",
       "      <path clip-path=\"url(#p923a6f6e7e)\" d=\"M 51.378125 65.283288 \r\n",
       "L 246.678125 65.283288 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_16\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"51.378125\" xlink:href=\"#mb15c192f53\" y=\"65.283288\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_9\">\r\n",
       "      <!-- $\\mathdefault{10^{0}}$ -->\r\n",
       "      <g transform=\"translate(26.778125 69.082506)scale(0.1 -0.1)\">\r\n",
       "       <use transform=\"translate(0 0.765625)\" xlink:href=\"#DejaVuSans-31\"/>\r\n",
       "       <use transform=\"translate(63.623047 0.765625)\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use transform=\"translate(128.203125 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_4\">\r\n",
       "     <g id=\"line2d_17\">\r\n",
       "      <path clip-path=\"url(#p923a6f6e7e)\" d=\"M 51.378125 38.141253 \r\n",
       "L 246.678125 38.141253 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_18\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"51.378125\" xlink:href=\"#mb15c192f53\" y=\"38.141253\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_10\">\r\n",
       "      <!-- $\\mathdefault{10^{1}}$ -->\r\n",
       "      <g transform=\"translate(26.778125 41.940472)scale(0.1 -0.1)\">\r\n",
       "       <use transform=\"translate(0 0.684375)\" xlink:href=\"#DejaVuSans-31\"/>\r\n",
       "       <use transform=\"translate(63.623047 0.684375)\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use transform=\"translate(128.203125 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-31\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_5\">\r\n",
       "     <g id=\"line2d_19\">\r\n",
       "      <path clip-path=\"url(#p923a6f6e7e)\" d=\"M 51.378125 10.999219 \r\n",
       "L 246.678125 10.999219 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_20\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"51.378125\" xlink:href=\"#mb15c192f53\" y=\"10.999219\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_11\">\r\n",
       "      <!-- $\\mathdefault{10^{2}}$ -->\r\n",
       "      <g transform=\"translate(26.778125 14.798437)scale(0.1 -0.1)\">\r\n",
       "       <use transform=\"translate(0 0.765625)\" xlink:href=\"#DejaVuSans-31\"/>\r\n",
       "       <use transform=\"translate(63.623047 0.765625)\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use transform=\"translate(128.203125 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-32\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_6\">\r\n",
       "     <g id=\"line2d_21\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 0 0 \r\n",
       "L -2 0 \r\n",
       "\" id=\"ma0f059bbbd\" style=\"stroke:#000000;stroke-width:0.6;\"/>\r\n",
       "      </defs>\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#ma0f059bbbd\" y=\"138.538824\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_7\">\r\n",
       "     <g id=\"line2d_22\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#ma0f059bbbd\" y=\"133.759349\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_8\">\r\n",
       "     <g id=\"line2d_23\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#ma0f059bbbd\" y=\"130.368258\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_9\">\r\n",
       "     <g id=\"line2d_24\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#ma0f059bbbd\" y=\"127.737923\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_10\">\r\n",
       "     <g id=\"line2d_25\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#ma0f059bbbd\" y=\"125.588783\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_11\">\r\n",
       "     <g id=\"line2d_26\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#ma0f059bbbd\" y=\"123.771711\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_12\">\r\n",
       "     <g id=\"line2d_27\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#ma0f059bbbd\" y=\"122.197691\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_13\">\r\n",
       "     <g id=\"line2d_28\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#ma0f059bbbd\" y=\"120.809308\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_14\">\r\n",
       "     <g id=\"line2d_29\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#ma0f059bbbd\" y=\"111.39679\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_15\">\r\n",
       "     <g id=\"line2d_30\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#ma0f059bbbd\" y=\"106.617315\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_16\">\r\n",
       "     <g id=\"line2d_31\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#ma0f059bbbd\" y=\"103.226223\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_17\">\r\n",
       "     <g id=\"line2d_32\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#ma0f059bbbd\" y=\"100.595888\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_18\">\r\n",
       "     <g id=\"line2d_33\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#ma0f059bbbd\" y=\"98.446748\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_19\">\r\n",
       "     <g id=\"line2d_34\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#ma0f059bbbd\" y=\"96.629676\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_20\">\r\n",
       "     <g id=\"line2d_35\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#ma0f059bbbd\" y=\"95.055657\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_21\">\r\n",
       "     <g id=\"line2d_36\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#ma0f059bbbd\" y=\"93.667273\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_22\">\r\n",
       "     <g id=\"line2d_37\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#ma0f059bbbd\" y=\"84.254755\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_23\">\r\n",
       "     <g id=\"line2d_38\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#ma0f059bbbd\" y=\"79.47528\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_24\">\r\n",
       "     <g id=\"line2d_39\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#ma0f059bbbd\" y=\"76.084189\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_25\">\r\n",
       "     <g id=\"line2d_40\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#ma0f059bbbd\" y=\"73.453854\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_26\">\r\n",
       "     <g id=\"line2d_41\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#ma0f059bbbd\" y=\"71.304714\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_27\">\r\n",
       "     <g id=\"line2d_42\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#ma0f059bbbd\" y=\"69.487642\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_28\">\r\n",
       "     <g id=\"line2d_43\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#ma0f059bbbd\" y=\"67.913622\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_29\">\r\n",
       "     <g id=\"line2d_44\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#ma0f059bbbd\" y=\"66.525239\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_30\">\r\n",
       "     <g id=\"line2d_45\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#ma0f059bbbd\" y=\"57.112721\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_31\">\r\n",
       "     <g id=\"line2d_46\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#ma0f059bbbd\" y=\"52.333246\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_32\">\r\n",
       "     <g id=\"line2d_47\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#ma0f059bbbd\" y=\"48.942155\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_33\">\r\n",
       "     <g id=\"line2d_48\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#ma0f059bbbd\" y=\"46.31182\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_34\">\r\n",
       "     <g id=\"line2d_49\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#ma0f059bbbd\" y=\"44.16268\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_35\">\r\n",
       "     <g id=\"line2d_50\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#ma0f059bbbd\" y=\"42.345607\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_36\">\r\n",
       "     <g id=\"line2d_51\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#ma0f059bbbd\" y=\"40.771588\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_37\">\r\n",
       "     <g id=\"line2d_52\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#ma0f059bbbd\" y=\"39.383205\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_38\">\r\n",
       "     <g id=\"line2d_53\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#ma0f059bbbd\" y=\"29.970687\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_39\">\r\n",
       "     <g id=\"line2d_54\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#ma0f059bbbd\" y=\"25.191212\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_40\">\r\n",
       "     <g id=\"line2d_55\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#ma0f059bbbd\" y=\"21.80012\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_41\">\r\n",
       "     <g id=\"line2d_56\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#ma0f059bbbd\" y=\"19.169785\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_42\">\r\n",
       "     <g id=\"line2d_57\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#ma0f059bbbd\" y=\"17.020645\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_43\">\r\n",
       "     <g id=\"line2d_58\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#ma0f059bbbd\" y=\"15.203573\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_44\">\r\n",
       "     <g id=\"line2d_59\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#ma0f059bbbd\" y=\"13.629554\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_45\">\r\n",
       "     <g id=\"line2d_60\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#ma0f059bbbd\" y=\"12.24117\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"text_12\">\r\n",
       "     <!-- loss -->\r\n",
       "     <g transform=\"translate(14.798437 88.098928)rotate(-90)scale(0.1 -0.1)\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 603 4863 \r\n",
       "L 1178 4863 \r\n",
       "L 1178 0 \r\n",
       "L 603 0 \r\n",
       "L 603 4863 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-6c\" transform=\"scale(0.015625)\"/>\r\n",
       "      </defs>\r\n",
       "      <use xlink:href=\"#DejaVuSans-6c\"/>\r\n",
       "      <use x=\"27.783203\" xlink:href=\"#DejaVuSans-6f\"/>\r\n",
       "      <use x=\"88.964844\" xlink:href=\"#DejaVuSans-73\"/>\r\n",
       "      <use x=\"141.064453\" xlink:href=\"#DejaVuSans-73\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "   <g id=\"line2d_61\">\r\n",
       "    <path clip-path=\"url(#p923a6f6e7e)\" d=\"M 51.378125 28.089718 \r\n",
       "L 61.657072 49.2394 \r\n",
       "L 71.93602 68.267673 \r\n",
       "L 82.214967 85.572271 \r\n",
       "L 92.493914 101.352631 \r\n",
       "L 102.772862 115.105708 \r\n",
       "L 113.051809 125.685282 \r\n",
       "L 123.330757 132.115602 \r\n",
       "L 133.609704 135.321995 \r\n",
       "L 143.888651 136.799285 \r\n",
       "L 154.167599 137.594478 \r\n",
       "L 164.446546 138.228562 \r\n",
       "L 174.725493 138.5973 \r\n",
       "L 185.004441 138.86528 \r\n",
       "L 195.283388 139.033889 \r\n",
       "L 205.562336 139.243484 \r\n",
       "L 215.841283 139.550453 \r\n",
       "L 226.12023 139.78142 \r\n",
       "L 236.399178 140.06148 \r\n",
       "L 246.678125 140.213843 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-linecap:square;stroke-width:1.5;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"line2d_62\">\r\n",
       "    <path clip-path=\"url(#p923a6f6e7e)\" d=\"M 51.378125 16.668389 \r\n",
       "L 61.657072 21.470422 \r\n",
       "L 71.93602 25.912791 \r\n",
       "L 82.214967 30.240036 \r\n",
       "L 92.493914 34.522767 \r\n",
       "L 102.772862 38.78745 \r\n",
       "L 113.051809 43.043128 \r\n",
       "L 123.330757 47.292301 \r\n",
       "L 133.609704 51.534255 \r\n",
       "L 143.888651 55.766849 \r\n",
       "L 154.167599 59.987295 \r\n",
       "L 164.446546 64.189911 \r\n",
       "L 174.725493 68.36722 \r\n",
       "L 185.004441 72.508977 \r\n",
       "L 195.283388 76.599975 \r\n",
       "L 205.562336 80.623775 \r\n",
       "L 215.841283 84.553752 \r\n",
       "L 226.12023 88.359736 \r\n",
       "L 236.399178 92.002034 \r\n",
       "L 246.678125 95.437348 \r\n",
       "\" style=\"fill:none;stroke:#bf00bf;stroke-dasharray:5.55,2.4;stroke-dashoffset:0;stroke-width:1.5;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_3\">\r\n",
       "    <path d=\"M 51.378125 146.391116 \r\n",
       "L 51.378125 10.491116 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_4\">\r\n",
       "    <path d=\"M 246.678125 146.391116 \r\n",
       "L 246.678125 10.491116 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_5\">\r\n",
       "    <path d=\"M 51.378125 146.391116 \r\n",
       "L 246.678125 146.391116 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_6\">\r\n",
       "    <path d=\"M 51.378125 10.491116 \r\n",
       "L 246.678125 10.491116 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"legend_1\">\r\n",
       "    <g id=\"patch_7\">\r\n",
       "     <path d=\"M 184.403125 47.847366 \r\n",
       "L 239.678125 47.847366 \r\n",
       "Q 241.678125 47.847366 241.678125 45.847366 \r\n",
       "L 241.678125 17.491116 \r\n",
       "Q 241.678125 15.491116 239.678125 15.491116 \r\n",
       "L 184.403125 15.491116 \r\n",
       "Q 182.403125 15.491116 182.403125 17.491116 \r\n",
       "L 182.403125 45.847366 \r\n",
       "Q 182.403125 47.847366 184.403125 47.847366 \r\n",
       "z\r\n",
       "\" style=\"fill:#ffffff;opacity:0.8;stroke:#cccccc;stroke-linejoin:miter;\"/>\r\n",
       "    </g>\r\n",
       "    <g id=\"line2d_63\">\r\n",
       "     <path d=\"M 186.403125 23.589553 \r\n",
       "L 206.403125 23.589553 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-linecap:square;stroke-width:1.5;\"/>\r\n",
       "    </g>\r\n",
       "    <g id=\"line2d_64\"/>\r\n",
       "    <g id=\"text_13\">\r\n",
       "     <!-- train -->\r\n",
       "     <g transform=\"translate(214.403125 27.089553)scale(0.1 -0.1)\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 1172 4494 \r\n",
       "L 1172 3500 \r\n",
       "L 2356 3500 \r\n",
       "L 2356 3053 \r\n",
       "L 1172 3053 \r\n",
       "L 1172 1153 \r\n",
       "Q 1172 725 1289 603 \r\n",
       "Q 1406 481 1766 481 \r\n",
       "L 2356 481 \r\n",
       "L 2356 0 \r\n",
       "L 1766 0 \r\n",
       "Q 1100 0 847 248 \r\n",
       "Q 594 497 594 1153 \r\n",
       "L 594 3053 \r\n",
       "L 172 3053 \r\n",
       "L 172 3500 \r\n",
       "L 594 3500 \r\n",
       "L 594 4494 \r\n",
       "L 1172 4494 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-74\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 2631 2963 \r\n",
       "Q 2534 3019 2420 3045 \r\n",
       "Q 2306 3072 2169 3072 \r\n",
       "Q 1681 3072 1420 2755 \r\n",
       "Q 1159 2438 1159 1844 \r\n",
       "L 1159 0 \r\n",
       "L 581 0 \r\n",
       "L 581 3500 \r\n",
       "L 1159 3500 \r\n",
       "L 1159 2956 \r\n",
       "Q 1341 3275 1631 3429 \r\n",
       "Q 1922 3584 2338 3584 \r\n",
       "Q 2397 3584 2469 3576 \r\n",
       "Q 2541 3569 2628 3553 \r\n",
       "L 2631 2963 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-72\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 2194 1759 \r\n",
       "Q 1497 1759 1228 1600 \r\n",
       "Q 959 1441 959 1056 \r\n",
       "Q 959 750 1161 570 \r\n",
       "Q 1363 391 1709 391 \r\n",
       "Q 2188 391 2477 730 \r\n",
       "Q 2766 1069 2766 1631 \r\n",
       "L 2766 1759 \r\n",
       "L 2194 1759 \r\n",
       "z\r\n",
       "M 3341 1997 \r\n",
       "L 3341 0 \r\n",
       "L 2766 0 \r\n",
       "L 2766 531 \r\n",
       "Q 2569 213 2275 61 \r\n",
       "Q 1981 -91 1556 -91 \r\n",
       "Q 1019 -91 701 211 \r\n",
       "Q 384 513 384 1019 \r\n",
       "Q 384 1609 779 1909 \r\n",
       "Q 1175 2209 1959 2209 \r\n",
       "L 2766 2209 \r\n",
       "L 2766 2266 \r\n",
       "Q 2766 2663 2505 2880 \r\n",
       "Q 2244 3097 1772 3097 \r\n",
       "Q 1472 3097 1187 3025 \r\n",
       "Q 903 2953 641 2809 \r\n",
       "L 641 3341 \r\n",
       "Q 956 3463 1253 3523 \r\n",
       "Q 1550 3584 1831 3584 \r\n",
       "Q 2591 3584 2966 3190 \r\n",
       "Q 3341 2797 3341 1997 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-61\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 603 3500 \r\n",
       "L 1178 3500 \r\n",
       "L 1178 0 \r\n",
       "L 603 0 \r\n",
       "L 603 3500 \r\n",
       "z\r\n",
       "M 603 4863 \r\n",
       "L 1178 4863 \r\n",
       "L 1178 4134 \r\n",
       "L 603 4134 \r\n",
       "L 603 4863 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-69\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 3513 2113 \r\n",
       "L 3513 0 \r\n",
       "L 2938 0 \r\n",
       "L 2938 2094 \r\n",
       "Q 2938 2591 2744 2837 \r\n",
       "Q 2550 3084 2163 3084 \r\n",
       "Q 1697 3084 1428 2787 \r\n",
       "Q 1159 2491 1159 1978 \r\n",
       "L 1159 0 \r\n",
       "L 581 0 \r\n",
       "L 581 3500 \r\n",
       "L 1159 3500 \r\n",
       "L 1159 2956 \r\n",
       "Q 1366 3272 1645 3428 \r\n",
       "Q 1925 3584 2291 3584 \r\n",
       "Q 2894 3584 3203 3211 \r\n",
       "Q 3513 2838 3513 2113 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-6e\" transform=\"scale(0.015625)\"/>\r\n",
       "      </defs>\r\n",
       "      <use xlink:href=\"#DejaVuSans-74\"/>\r\n",
       "      <use x=\"39.208984\" xlink:href=\"#DejaVuSans-72\"/>\r\n",
       "      <use x=\"80.322266\" xlink:href=\"#DejaVuSans-61\"/>\r\n",
       "      <use x=\"141.601562\" xlink:href=\"#DejaVuSans-69\"/>\r\n",
       "      <use x=\"169.384766\" xlink:href=\"#DejaVuSans-6e\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"line2d_65\">\r\n",
       "     <path d=\"M 186.403125 38.267678 \r\n",
       "L 206.403125 38.267678 \r\n",
       "\" style=\"fill:none;stroke:#bf00bf;stroke-dasharray:5.55,2.4;stroke-dashoffset:0;stroke-width:1.5;\"/>\r\n",
       "    </g>\r\n",
       "    <g id=\"line2d_66\"/>\r\n",
       "    <g id=\"text_14\">\r\n",
       "     <!-- test -->\r\n",
       "     <g transform=\"translate(214.403125 41.767678)scale(0.1 -0.1)\">\r\n",
       "      <use xlink:href=\"#DejaVuSans-74\"/>\r\n",
       "      <use x=\"39.208984\" xlink:href=\"#DejaVuSans-65\"/>\r\n",
       "      <use x=\"100.732422\" xlink:href=\"#DejaVuSans-73\"/>\r\n",
       "      <use x=\"152.832031\" xlink:href=\"#DejaVuSans-74\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "  </g>\r\n",
       " </g>\r\n",
       " <defs>\r\n",
       "  <clipPath id=\"p923a6f6e7e\">\r\n",
       "   <rect height=\"135.9\" width=\"195.3\" x=\"51.378125\" y=\"10.491116\"/>\r\n",
       "  </clipPath>\r\n",
       " </defs>\r\n",
       "</svg>\r\n"
      ],
      "text/plain": [
       "<Figure size 252x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "train(lambd=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 22
   },
   "source": [
    "## [**简洁实现**]\n",
    "\n",
    "由于权重衰减在神经网络优化中很常用，深度学习框架为了便于使用权重衰减，便将权重衰减集成到优化算法中，以便与任何损失函数结合使用。此外，这种集成还有计算上的好处，允许在不增加任何额外的计算开销的情况下向算法中添加权重衰减。由于更新的权重衰减部分仅依赖于每个参数的当前值，因此优化器必须至少接触每个参数一次。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 24,
    "tab": [
     "pytorch"
    ]
   },
   "source": [
    "在下面的代码中，我们在实例化优化器时直接通过`weight_decay`指定weight decay超参数。默认情况下，PyTorch同时衰减权重和偏移。这里我们只为权重设置了`weight_decay`，所以bias参数$b$不会衰减。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "origin_pos": 27,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [],
   "source": [
    "def train_concise(wd):\n",
    "    weight_attr = paddle.framework.ParamAttr(initializer=paddle.nn.initializer.Normal(mean=0.0, std=1.0))\n",
    "    bias_attr = paddle.framework.ParamAttr(initializer=paddle.nn.initializer.Normal(mean=0.0, std=1.0))\n",
    "    net = nn.Sequential(nn.Linear(num_inputs, 1 ,weight_attr=weight_attr, bias_attr=bias_attr))\n",
    "    loss = nn.MSELoss()\n",
    "    num_epochs, lr = 100, 0.003\n",
    "    # 偏置参数没有衰减。\n",
    "    trainer = paddle.optimizer.SGD(parameters = net[0].parameters(), learning_rate=lr,weight_decay = wd*1.0)\n",
    "    animator = d2l.Animator(xlabel='epochs', ylabel='loss', yscale='log',\n",
    "                            xlim=[5, num_epochs], legend=['train', 'test'])\n",
    "    for epoch in range(num_epochs):\n",
    "        for X, y in train_iter:\n",
    "            l = loss(net(X), y)\n",
    "            l.backward()\n",
    "            trainer.step()\n",
    "            trainer.clear_grad()\n",
    "        if (epoch + 1) % 5 == 0:\n",
    "            animator.add(epoch + 1, (d2l.evaluate_loss(net, train_iter, loss),\n",
    "                                     d2l.evaluate_loss(net, test_iter, loss)))\n",
    "    print('w的L2范数：', net[0].weight.norm().item())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 29
   },
   "source": [
    "[**这些图看起来和我们从零开始实现权重衰减时的图相同**]。然而，它们运行得更快，更容易实现，对于更复杂的问题，这一好处将变得更加明显。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "origin_pos": 30,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "w的L2范数： 13.050535202026367\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\r\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\r\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\r\n",
       "<svg height=\"180.65625pt\" version=\"1.1\" viewBox=\"0 0 263.421875 180.65625\" width=\"263.421875pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\r\n",
       " <metadata>\r\n",
       "  <rdf:RDF xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\r\n",
       "   <cc:Work>\r\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\r\n",
       "    <dc:date>2021-09-12T16:43:08.244801</dc:date>\r\n",
       "    <dc:format>image/svg+xml</dc:format>\r\n",
       "    <dc:creator>\r\n",
       "     <cc:Agent>\r\n",
       "      <dc:title>Matplotlib v3.4.3, https://matplotlib.org/</dc:title>\r\n",
       "     </cc:Agent>\r\n",
       "    </dc:creator>\r\n",
       "   </cc:Work>\r\n",
       "  </rdf:RDF>\r\n",
       " </metadata>\r\n",
       " <defs>\r\n",
       "  <style type=\"text/css\">*{stroke-linecap:butt;stroke-linejoin:round;}</style>\r\n",
       " </defs>\r\n",
       " <g id=\"figure_1\">\r\n",
       "  <g id=\"patch_1\">\r\n",
       "   <path d=\"M 0 180.65625 \r\n",
       "L 263.421875 180.65625 \r\n",
       "L 263.421875 0 \r\n",
       "L 0 0 \r\n",
       "z\r\n",
       "\" style=\"fill:none;\"/>\r\n",
       "  </g>\r\n",
       "  <g id=\"axes_1\">\r\n",
       "   <g id=\"patch_2\">\r\n",
       "    <path d=\"M 51.378125 143.1 \r\n",
       "L 246.678125 143.1 \r\n",
       "L 246.678125 7.2 \r\n",
       "L 51.378125 7.2 \r\n",
       "z\r\n",
       "\" style=\"fill:#ffffff;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"matplotlib.axis_1\">\r\n",
       "    <g id=\"xtick_1\">\r\n",
       "     <g id=\"line2d_1\">\r\n",
       "      <path clip-path=\"url(#pf89746b9ad)\" d=\"M 82.214967 143.1 \r\n",
       "L 82.214967 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_2\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 0 0 \r\n",
       "L 0 3.5 \r\n",
       "\" id=\"m3c7cfdc020\" style=\"stroke:#000000;stroke-width:0.8;\"/>\r\n",
       "      </defs>\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"82.214967\" xlink:href=\"#m3c7cfdc020\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_1\">\r\n",
       "      <!-- 20 -->\r\n",
       "      <g transform=\"translate(75.852467 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 1228 531 \r\n",
       "L 3431 531 \r\n",
       "L 3431 0 \r\n",
       "L 469 0 \r\n",
       "L 469 531 \r\n",
       "Q 828 903 1448 1529 \r\n",
       "Q 2069 2156 2228 2338 \r\n",
       "Q 2531 2678 2651 2914 \r\n",
       "Q 2772 3150 2772 3378 \r\n",
       "Q 2772 3750 2511 3984 \r\n",
       "Q 2250 4219 1831 4219 \r\n",
       "Q 1534 4219 1204 4116 \r\n",
       "Q 875 4013 500 3803 \r\n",
       "L 500 4441 \r\n",
       "Q 881 4594 1212 4672 \r\n",
       "Q 1544 4750 1819 4750 \r\n",
       "Q 2544 4750 2975 4387 \r\n",
       "Q 3406 4025 3406 3419 \r\n",
       "Q 3406 3131 3298 2873 \r\n",
       "Q 3191 2616 2906 2266 \r\n",
       "Q 2828 2175 2409 1742 \r\n",
       "Q 1991 1309 1228 531 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-32\" transform=\"scale(0.015625)\"/>\r\n",
       "        <path d=\"M 2034 4250 \r\n",
       "Q 1547 4250 1301 3770 \r\n",
       "Q 1056 3291 1056 2328 \r\n",
       "Q 1056 1369 1301 889 \r\n",
       "Q 1547 409 2034 409 \r\n",
       "Q 2525 409 2770 889 \r\n",
       "Q 3016 1369 3016 2328 \r\n",
       "Q 3016 3291 2770 3770 \r\n",
       "Q 2525 4250 2034 4250 \r\n",
       "z\r\n",
       "M 2034 4750 \r\n",
       "Q 2819 4750 3233 4129 \r\n",
       "Q 3647 3509 3647 2328 \r\n",
       "Q 3647 1150 3233 529 \r\n",
       "Q 2819 -91 2034 -91 \r\n",
       "Q 1250 -91 836 529 \r\n",
       "Q 422 1150 422 2328 \r\n",
       "Q 422 3509 836 4129 \r\n",
       "Q 1250 4750 2034 4750 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-30\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_2\">\r\n",
       "     <g id=\"line2d_3\">\r\n",
       "      <path clip-path=\"url(#pf89746b9ad)\" d=\"M 123.330757 143.1 \r\n",
       "L 123.330757 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_4\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"123.330757\" xlink:href=\"#m3c7cfdc020\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_2\">\r\n",
       "      <!-- 40 -->\r\n",
       "      <g transform=\"translate(116.968257 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 2419 4116 \r\n",
       "L 825 1625 \r\n",
       "L 2419 1625 \r\n",
       "L 2419 4116 \r\n",
       "z\r\n",
       "M 2253 4666 \r\n",
       "L 3047 4666 \r\n",
       "L 3047 1625 \r\n",
       "L 3713 1625 \r\n",
       "L 3713 1100 \r\n",
       "L 3047 1100 \r\n",
       "L 3047 0 \r\n",
       "L 2419 0 \r\n",
       "L 2419 1100 \r\n",
       "L 313 1100 \r\n",
       "L 313 1709 \r\n",
       "L 2253 4666 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-34\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-34\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_3\">\r\n",
       "     <g id=\"line2d_5\">\r\n",
       "      <path clip-path=\"url(#pf89746b9ad)\" d=\"M 164.446546 143.1 \r\n",
       "L 164.446546 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_6\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"164.446546\" xlink:href=\"#m3c7cfdc020\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_3\">\r\n",
       "      <!-- 60 -->\r\n",
       "      <g transform=\"translate(158.084046 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 2113 2584 \r\n",
       "Q 1688 2584 1439 2293 \r\n",
       "Q 1191 2003 1191 1497 \r\n",
       "Q 1191 994 1439 701 \r\n",
       "Q 1688 409 2113 409 \r\n",
       "Q 2538 409 2786 701 \r\n",
       "Q 3034 994 3034 1497 \r\n",
       "Q 3034 2003 2786 2293 \r\n",
       "Q 2538 2584 2113 2584 \r\n",
       "z\r\n",
       "M 3366 4563 \r\n",
       "L 3366 3988 \r\n",
       "Q 3128 4100 2886 4159 \r\n",
       "Q 2644 4219 2406 4219 \r\n",
       "Q 1781 4219 1451 3797 \r\n",
       "Q 1122 3375 1075 2522 \r\n",
       "Q 1259 2794 1537 2939 \r\n",
       "Q 1816 3084 2150 3084 \r\n",
       "Q 2853 3084 3261 2657 \r\n",
       "Q 3669 2231 3669 1497 \r\n",
       "Q 3669 778 3244 343 \r\n",
       "Q 2819 -91 2113 -91 \r\n",
       "Q 1303 -91 875 529 \r\n",
       "Q 447 1150 447 2328 \r\n",
       "Q 447 3434 972 4092 \r\n",
       "Q 1497 4750 2381 4750 \r\n",
       "Q 2619 4750 2861 4703 \r\n",
       "Q 3103 4656 3366 4563 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-36\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-36\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_4\">\r\n",
       "     <g id=\"line2d_7\">\r\n",
       "      <path clip-path=\"url(#pf89746b9ad)\" d=\"M 205.562336 143.1 \r\n",
       "L 205.562336 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_8\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"205.562336\" xlink:href=\"#m3c7cfdc020\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_4\">\r\n",
       "      <!-- 80 -->\r\n",
       "      <g transform=\"translate(199.199836 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 2034 2216 \r\n",
       "Q 1584 2216 1326 1975 \r\n",
       "Q 1069 1734 1069 1313 \r\n",
       "Q 1069 891 1326 650 \r\n",
       "Q 1584 409 2034 409 \r\n",
       "Q 2484 409 2743 651 \r\n",
       "Q 3003 894 3003 1313 \r\n",
       "Q 3003 1734 2745 1975 \r\n",
       "Q 2488 2216 2034 2216 \r\n",
       "z\r\n",
       "M 1403 2484 \r\n",
       "Q 997 2584 770 2862 \r\n",
       "Q 544 3141 544 3541 \r\n",
       "Q 544 4100 942 4425 \r\n",
       "Q 1341 4750 2034 4750 \r\n",
       "Q 2731 4750 3128 4425 \r\n",
       "Q 3525 4100 3525 3541 \r\n",
       "Q 3525 3141 3298 2862 \r\n",
       "Q 3072 2584 2669 2484 \r\n",
       "Q 3125 2378 3379 2068 \r\n",
       "Q 3634 1759 3634 1313 \r\n",
       "Q 3634 634 3220 271 \r\n",
       "Q 2806 -91 2034 -91 \r\n",
       "Q 1263 -91 848 271 \r\n",
       "Q 434 634 434 1313 \r\n",
       "Q 434 1759 690 2068 \r\n",
       "Q 947 2378 1403 2484 \r\n",
       "z\r\n",
       "M 1172 3481 \r\n",
       "Q 1172 3119 1398 2916 \r\n",
       "Q 1625 2713 2034 2713 \r\n",
       "Q 2441 2713 2670 2916 \r\n",
       "Q 2900 3119 2900 3481 \r\n",
       "Q 2900 3844 2670 4047 \r\n",
       "Q 2441 4250 2034 4250 \r\n",
       "Q 1625 4250 1398 4047 \r\n",
       "Q 1172 3844 1172 3481 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-38\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-38\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_5\">\r\n",
       "     <g id=\"line2d_9\">\r\n",
       "      <path clip-path=\"url(#pf89746b9ad)\" d=\"M 246.678125 143.1 \r\n",
       "L 246.678125 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_10\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"246.678125\" xlink:href=\"#m3c7cfdc020\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_5\">\r\n",
       "      <!-- 100 -->\r\n",
       "      <g transform=\"translate(237.134375 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 794 531 \r\n",
       "L 1825 531 \r\n",
       "L 1825 4091 \r\n",
       "L 703 3866 \r\n",
       "L 703 4441 \r\n",
       "L 1819 4666 \r\n",
       "L 2450 4666 \r\n",
       "L 2450 531 \r\n",
       "L 3481 531 \r\n",
       "L 3481 0 \r\n",
       "L 794 0 \r\n",
       "L 794 531 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-31\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"text_6\">\r\n",
       "     <!-- epochs -->\r\n",
       "     <g transform=\"translate(131.195312 171.376563)scale(0.1 -0.1)\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 3597 1894 \r\n",
       "L 3597 1613 \r\n",
       "L 953 1613 \r\n",
       "Q 991 1019 1311 708 \r\n",
       "Q 1631 397 2203 397 \r\n",
       "Q 2534 397 2845 478 \r\n",
       "Q 3156 559 3463 722 \r\n",
       "L 3463 178 \r\n",
       "Q 3153 47 2828 -22 \r\n",
       "Q 2503 -91 2169 -91 \r\n",
       "Q 1331 -91 842 396 \r\n",
       "Q 353 884 353 1716 \r\n",
       "Q 353 2575 817 3079 \r\n",
       "Q 1281 3584 2069 3584 \r\n",
       "Q 2775 3584 3186 3129 \r\n",
       "Q 3597 2675 3597 1894 \r\n",
       "z\r\n",
       "M 3022 2063 \r\n",
       "Q 3016 2534 2758 2815 \r\n",
       "Q 2500 3097 2075 3097 \r\n",
       "Q 1594 3097 1305 2825 \r\n",
       "Q 1016 2553 972 2059 \r\n",
       "L 3022 2063 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-65\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 1159 525 \r\n",
       "L 1159 -1331 \r\n",
       "L 581 -1331 \r\n",
       "L 581 3500 \r\n",
       "L 1159 3500 \r\n",
       "L 1159 2969 \r\n",
       "Q 1341 3281 1617 3432 \r\n",
       "Q 1894 3584 2278 3584 \r\n",
       "Q 2916 3584 3314 3078 \r\n",
       "Q 3713 2572 3713 1747 \r\n",
       "Q 3713 922 3314 415 \r\n",
       "Q 2916 -91 2278 -91 \r\n",
       "Q 1894 -91 1617 61 \r\n",
       "Q 1341 213 1159 525 \r\n",
       "z\r\n",
       "M 3116 1747 \r\n",
       "Q 3116 2381 2855 2742 \r\n",
       "Q 2594 3103 2138 3103 \r\n",
       "Q 1681 3103 1420 2742 \r\n",
       "Q 1159 2381 1159 1747 \r\n",
       "Q 1159 1113 1420 752 \r\n",
       "Q 1681 391 2138 391 \r\n",
       "Q 2594 391 2855 752 \r\n",
       "Q 3116 1113 3116 1747 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-70\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 1959 3097 \r\n",
       "Q 1497 3097 1228 2736 \r\n",
       "Q 959 2375 959 1747 \r\n",
       "Q 959 1119 1226 758 \r\n",
       "Q 1494 397 1959 397 \r\n",
       "Q 2419 397 2687 759 \r\n",
       "Q 2956 1122 2956 1747 \r\n",
       "Q 2956 2369 2687 2733 \r\n",
       "Q 2419 3097 1959 3097 \r\n",
       "z\r\n",
       "M 1959 3584 \r\n",
       "Q 2709 3584 3137 3096 \r\n",
       "Q 3566 2609 3566 1747 \r\n",
       "Q 3566 888 3137 398 \r\n",
       "Q 2709 -91 1959 -91 \r\n",
       "Q 1206 -91 779 398 \r\n",
       "Q 353 888 353 1747 \r\n",
       "Q 353 2609 779 3096 \r\n",
       "Q 1206 3584 1959 3584 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-6f\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 3122 3366 \r\n",
       "L 3122 2828 \r\n",
       "Q 2878 2963 2633 3030 \r\n",
       "Q 2388 3097 2138 3097 \r\n",
       "Q 1578 3097 1268 2742 \r\n",
       "Q 959 2388 959 1747 \r\n",
       "Q 959 1106 1268 751 \r\n",
       "Q 1578 397 2138 397 \r\n",
       "Q 2388 397 2633 464 \r\n",
       "Q 2878 531 3122 666 \r\n",
       "L 3122 134 \r\n",
       "Q 2881 22 2623 -34 \r\n",
       "Q 2366 -91 2075 -91 \r\n",
       "Q 1284 -91 818 406 \r\n",
       "Q 353 903 353 1747 \r\n",
       "Q 353 2603 823 3093 \r\n",
       "Q 1294 3584 2113 3584 \r\n",
       "Q 2378 3584 2631 3529 \r\n",
       "Q 2884 3475 3122 3366 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-63\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 3513 2113 \r\n",
       "L 3513 0 \r\n",
       "L 2938 0 \r\n",
       "L 2938 2094 \r\n",
       "Q 2938 2591 2744 2837 \r\n",
       "Q 2550 3084 2163 3084 \r\n",
       "Q 1697 3084 1428 2787 \r\n",
       "Q 1159 2491 1159 1978 \r\n",
       "L 1159 0 \r\n",
       "L 581 0 \r\n",
       "L 581 4863 \r\n",
       "L 1159 4863 \r\n",
       "L 1159 2956 \r\n",
       "Q 1366 3272 1645 3428 \r\n",
       "Q 1925 3584 2291 3584 \r\n",
       "Q 2894 3584 3203 3211 \r\n",
       "Q 3513 2838 3513 2113 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-68\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 2834 3397 \r\n",
       "L 2834 2853 \r\n",
       "Q 2591 2978 2328 3040 \r\n",
       "Q 2066 3103 1784 3103 \r\n",
       "Q 1356 3103 1142 2972 \r\n",
       "Q 928 2841 928 2578 \r\n",
       "Q 928 2378 1081 2264 \r\n",
       "Q 1234 2150 1697 2047 \r\n",
       "L 1894 2003 \r\n",
       "Q 2506 1872 2764 1633 \r\n",
       "Q 3022 1394 3022 966 \r\n",
       "Q 3022 478 2636 193 \r\n",
       "Q 2250 -91 1575 -91 \r\n",
       "Q 1294 -91 989 -36 \r\n",
       "Q 684 19 347 128 \r\n",
       "L 347 722 \r\n",
       "Q 666 556 975 473 \r\n",
       "Q 1284 391 1588 391 \r\n",
       "Q 1994 391 2212 530 \r\n",
       "Q 2431 669 2431 922 \r\n",
       "Q 2431 1156 2273 1281 \r\n",
       "Q 2116 1406 1581 1522 \r\n",
       "L 1381 1569 \r\n",
       "Q 847 1681 609 1914 \r\n",
       "Q 372 2147 372 2553 \r\n",
       "Q 372 3047 722 3315 \r\n",
       "Q 1072 3584 1716 3584 \r\n",
       "Q 2034 3584 2315 3537 \r\n",
       "Q 2597 3491 2834 3397 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-73\" transform=\"scale(0.015625)\"/>\r\n",
       "      </defs>\r\n",
       "      <use xlink:href=\"#DejaVuSans-65\"/>\r\n",
       "      <use x=\"61.523438\" xlink:href=\"#DejaVuSans-70\"/>\r\n",
       "      <use x=\"125\" xlink:href=\"#DejaVuSans-6f\"/>\r\n",
       "      <use x=\"186.181641\" xlink:href=\"#DejaVuSans-63\"/>\r\n",
       "      <use x=\"241.162109\" xlink:href=\"#DejaVuSans-68\"/>\r\n",
       "      <use x=\"304.541016\" xlink:href=\"#DejaVuSans-73\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "   <g id=\"matplotlib.axis_2\">\r\n",
       "    <g id=\"ytick_1\">\r\n",
       "     <g id=\"line2d_11\">\r\n",
       "      <path clip-path=\"url(#pf89746b9ad)\" d=\"M 51.378125 121.549137 \r\n",
       "L 246.678125 121.549137 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_12\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 0 0 \r\n",
       "L -3.5 0 \r\n",
       "\" id=\"mf7e88dc907\" style=\"stroke:#000000;stroke-width:0.8;\"/>\r\n",
       "      </defs>\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"51.378125\" xlink:href=\"#mf7e88dc907\" y=\"121.549137\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_7\">\r\n",
       "      <!-- $\\mathdefault{10^{-9}}$ -->\r\n",
       "      <g transform=\"translate(20.878125 125.348356)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 678 2272 \r\n",
       "L 4684 2272 \r\n",
       "L 4684 1741 \r\n",
       "L 678 1741 \r\n",
       "L 678 2272 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-2212\" transform=\"scale(0.015625)\"/>\r\n",
       "        <path d=\"M 703 97 \r\n",
       "L 703 672 \r\n",
       "Q 941 559 1184 500 \r\n",
       "Q 1428 441 1663 441 \r\n",
       "Q 2288 441 2617 861 \r\n",
       "Q 2947 1281 2994 2138 \r\n",
       "Q 2813 1869 2534 1725 \r\n",
       "Q 2256 1581 1919 1581 \r\n",
       "Q 1219 1581 811 2004 \r\n",
       "Q 403 2428 403 3163 \r\n",
       "Q 403 3881 828 4315 \r\n",
       "Q 1253 4750 1959 4750 \r\n",
       "Q 2769 4750 3195 4129 \r\n",
       "Q 3622 3509 3622 2328 \r\n",
       "Q 3622 1225 3098 567 \r\n",
       "Q 2575 -91 1691 -91 \r\n",
       "Q 1453 -91 1209 -44 \r\n",
       "Q 966 3 703 97 \r\n",
       "z\r\n",
       "M 1959 2075 \r\n",
       "Q 2384 2075 2632 2365 \r\n",
       "Q 2881 2656 2881 3163 \r\n",
       "Q 2881 3666 2632 3958 \r\n",
       "Q 2384 4250 1959 4250 \r\n",
       "Q 1534 4250 1286 3958 \r\n",
       "Q 1038 3666 1038 3163 \r\n",
       "Q 1038 2656 1286 2365 \r\n",
       "Q 1534 2075 1959 2075 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-39\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use transform=\"translate(0 0.765625)\" xlink:href=\"#DejaVuSans-31\"/>\r\n",
       "       <use transform=\"translate(63.623047 0.765625)\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use transform=\"translate(128.203125 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-2212\"/>\r\n",
       "       <use transform=\"translate(186.855469 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-39\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_2\">\r\n",
       "     <g id=\"line2d_13\">\r\n",
       "      <path clip-path=\"url(#pf89746b9ad)\" d=\"M 51.378125 92.83311 \r\n",
       "L 246.678125 92.83311 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_14\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"51.378125\" xlink:href=\"#mf7e88dc907\" y=\"92.83311\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_8\">\r\n",
       "      <!-- $\\mathdefault{10^{-6}}$ -->\r\n",
       "      <g transform=\"translate(20.878125 96.632329)scale(0.1 -0.1)\">\r\n",
       "       <use transform=\"translate(0 0.765625)\" xlink:href=\"#DejaVuSans-31\"/>\r\n",
       "       <use transform=\"translate(63.623047 0.765625)\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use transform=\"translate(128.203125 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-2212\"/>\r\n",
       "       <use transform=\"translate(186.855469 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-36\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_3\">\r\n",
       "     <g id=\"line2d_15\">\r\n",
       "      <path clip-path=\"url(#pf89746b9ad)\" d=\"M 51.378125 64.117082 \r\n",
       "L 246.678125 64.117082 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_16\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"51.378125\" xlink:href=\"#mf7e88dc907\" y=\"64.117082\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_9\">\r\n",
       "      <!-- $\\mathdefault{10^{-3}}$ -->\r\n",
       "      <g transform=\"translate(20.878125 67.916301)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 2597 2516 \r\n",
       "Q 3050 2419 3304 2112 \r\n",
       "Q 3559 1806 3559 1356 \r\n",
       "Q 3559 666 3084 287 \r\n",
       "Q 2609 -91 1734 -91 \r\n",
       "Q 1441 -91 1130 -33 \r\n",
       "Q 819 25 488 141 \r\n",
       "L 488 750 \r\n",
       "Q 750 597 1062 519 \r\n",
       "Q 1375 441 1716 441 \r\n",
       "Q 2309 441 2620 675 \r\n",
       "Q 2931 909 2931 1356 \r\n",
       "Q 2931 1769 2642 2001 \r\n",
       "Q 2353 2234 1838 2234 \r\n",
       "L 1294 2234 \r\n",
       "L 1294 2753 \r\n",
       "L 1863 2753 \r\n",
       "Q 2328 2753 2575 2939 \r\n",
       "Q 2822 3125 2822 3475 \r\n",
       "Q 2822 3834 2567 4026 \r\n",
       "Q 2313 4219 1838 4219 \r\n",
       "Q 1578 4219 1281 4162 \r\n",
       "Q 984 4106 628 3988 \r\n",
       "L 628 4550 \r\n",
       "Q 988 4650 1302 4700 \r\n",
       "Q 1616 4750 1894 4750 \r\n",
       "Q 2613 4750 3031 4423 \r\n",
       "Q 3450 4097 3450 3541 \r\n",
       "Q 3450 3153 3228 2886 \r\n",
       "Q 3006 2619 2597 2516 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-33\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use transform=\"translate(0 0.765625)\" xlink:href=\"#DejaVuSans-31\"/>\r\n",
       "       <use transform=\"translate(63.623047 0.765625)\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use transform=\"translate(128.203125 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-2212\"/>\r\n",
       "       <use transform=\"translate(186.855469 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-33\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_4\">\r\n",
       "     <g id=\"line2d_17\">\r\n",
       "      <path clip-path=\"url(#pf89746b9ad)\" d=\"M 51.378125 35.401055 \r\n",
       "L 246.678125 35.401055 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_18\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"51.378125\" xlink:href=\"#mf7e88dc907\" y=\"35.401055\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_10\">\r\n",
       "      <!-- $\\mathdefault{10^{0}}$ -->\r\n",
       "      <g transform=\"translate(26.778125 39.200274)scale(0.1 -0.1)\">\r\n",
       "       <use transform=\"translate(0 0.765625)\" xlink:href=\"#DejaVuSans-31\"/>\r\n",
       "       <use transform=\"translate(63.623047 0.765625)\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use transform=\"translate(128.203125 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"text_11\">\r\n",
       "     <!-- loss -->\r\n",
       "     <g transform=\"translate(14.798437 84.807812)rotate(-90)scale(0.1 -0.1)\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 603 4863 \r\n",
       "L 1178 4863 \r\n",
       "L 1178 0 \r\n",
       "L 603 0 \r\n",
       "L 603 4863 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-6c\" transform=\"scale(0.015625)\"/>\r\n",
       "      </defs>\r\n",
       "      <use xlink:href=\"#DejaVuSans-6c\"/>\r\n",
       "      <use x=\"27.783203\" xlink:href=\"#DejaVuSans-6f\"/>\r\n",
       "      <use x=\"88.964844\" xlink:href=\"#DejaVuSans-73\"/>\r\n",
       "      <use x=\"141.064453\" xlink:href=\"#DejaVuSans-73\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "   <g id=\"line2d_19\">\r\n",
       "    <path clip-path=\"url(#pf89746b9ad)\" d=\"M 51.378125 24.212659 \r\n",
       "L 61.657072 32.876652 \r\n",
       "L 71.93602 40.374731 \r\n",
       "L 82.214967 47.483841 \r\n",
       "L 92.493914 54.381498 \r\n",
       "L 102.772862 61.077001 \r\n",
       "L 113.051809 67.591259 \r\n",
       "L 123.330757 73.938683 \r\n",
       "L 133.609704 80.131112 \r\n",
       "L 143.888651 86.178085 \r\n",
       "L 154.167599 92.10143 \r\n",
       "L 164.446546 97.9155 \r\n",
       "L 174.725493 103.669529 \r\n",
       "L 185.004441 109.383199 \r\n",
       "L 195.283388 115.049142 \r\n",
       "L 205.562336 120.532403 \r\n",
       "L 215.841283 125.978136 \r\n",
       "L 226.12023 130.738385 \r\n",
       "L 236.399178 134.425706 \r\n",
       "L 246.678125 136.922727 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-linecap:square;stroke-width:1.5;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"line2d_20\">\r\n",
       "    <path clip-path=\"url(#pf89746b9ad)\" d=\"M 51.378125 13.377273 \r\n",
       "L 61.657072 13.4932 \r\n",
       "L 71.93602 13.526101 \r\n",
       "L 82.214967 13.536354 \r\n",
       "L 92.493914 13.539976 \r\n",
       "L 102.772862 13.541086 \r\n",
       "L 113.051809 13.541431 \r\n",
       "L 123.330757 13.541498 \r\n",
       "L 133.609704 13.541494 \r\n",
       "L 143.888651 13.541477 \r\n",
       "L 154.167599 13.541462 \r\n",
       "L 164.446546 13.541451 \r\n",
       "L 174.725493 13.541445 \r\n",
       "L 185.004441 13.541442 \r\n",
       "L 195.283388 13.541439 \r\n",
       "L 205.562336 13.541438 \r\n",
       "L 215.841283 13.541438 \r\n",
       "L 226.12023 13.541437 \r\n",
       "L 236.399178 13.541437 \r\n",
       "L 246.678125 13.541437 \r\n",
       "\" style=\"fill:none;stroke:#bf00bf;stroke-dasharray:5.55,2.4;stroke-dashoffset:0;stroke-width:1.5;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_3\">\r\n",
       "    <path d=\"M 51.378125 143.1 \r\n",
       "L 51.378125 7.2 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_4\">\r\n",
       "    <path d=\"M 246.678125 143.1 \r\n",
       "L 246.678125 7.2 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_5\">\r\n",
       "    <path d=\"M 51.378125 143.1 \r\n",
       "L 246.678125 143.1 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_6\">\r\n",
       "    <path d=\"M 51.378125 7.2 \r\n",
       "L 246.678125 7.2 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"legend_1\">\r\n",
       "    <g id=\"patch_7\">\r\n",
       "     <path d=\"M 58.378125 138.1 \r\n",
       "L 113.653125 138.1 \r\n",
       "Q 115.653125 138.1 115.653125 136.1 \r\n",
       "L 115.653125 107.74375 \r\n",
       "Q 115.653125 105.74375 113.653125 105.74375 \r\n",
       "L 58.378125 105.74375 \r\n",
       "Q 56.378125 105.74375 56.378125 107.74375 \r\n",
       "L 56.378125 136.1 \r\n",
       "Q 56.378125 138.1 58.378125 138.1 \r\n",
       "z\r\n",
       "\" style=\"fill:#ffffff;opacity:0.8;stroke:#cccccc;stroke-linejoin:miter;\"/>\r\n",
       "    </g>\r\n",
       "    <g id=\"line2d_21\">\r\n",
       "     <path d=\"M 60.378125 113.842187 \r\n",
       "L 80.378125 113.842187 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-linecap:square;stroke-width:1.5;\"/>\r\n",
       "    </g>\r\n",
       "    <g id=\"line2d_22\"/>\r\n",
       "    <g id=\"text_12\">\r\n",
       "     <!-- train -->\r\n",
       "     <g transform=\"translate(88.378125 117.342187)scale(0.1 -0.1)\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 1172 4494 \r\n",
       "L 1172 3500 \r\n",
       "L 2356 3500 \r\n",
       "L 2356 3053 \r\n",
       "L 1172 3053 \r\n",
       "L 1172 1153 \r\n",
       "Q 1172 725 1289 603 \r\n",
       "Q 1406 481 1766 481 \r\n",
       "L 2356 481 \r\n",
       "L 2356 0 \r\n",
       "L 1766 0 \r\n",
       "Q 1100 0 847 248 \r\n",
       "Q 594 497 594 1153 \r\n",
       "L 594 3053 \r\n",
       "L 172 3053 \r\n",
       "L 172 3500 \r\n",
       "L 594 3500 \r\n",
       "L 594 4494 \r\n",
       "L 1172 4494 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-74\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 2631 2963 \r\n",
       "Q 2534 3019 2420 3045 \r\n",
       "Q 2306 3072 2169 3072 \r\n",
       "Q 1681 3072 1420 2755 \r\n",
       "Q 1159 2438 1159 1844 \r\n",
       "L 1159 0 \r\n",
       "L 581 0 \r\n",
       "L 581 3500 \r\n",
       "L 1159 3500 \r\n",
       "L 1159 2956 \r\n",
       "Q 1341 3275 1631 3429 \r\n",
       "Q 1922 3584 2338 3584 \r\n",
       "Q 2397 3584 2469 3576 \r\n",
       "Q 2541 3569 2628 3553 \r\n",
       "L 2631 2963 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-72\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 2194 1759 \r\n",
       "Q 1497 1759 1228 1600 \r\n",
       "Q 959 1441 959 1056 \r\n",
       "Q 959 750 1161 570 \r\n",
       "Q 1363 391 1709 391 \r\n",
       "Q 2188 391 2477 730 \r\n",
       "Q 2766 1069 2766 1631 \r\n",
       "L 2766 1759 \r\n",
       "L 2194 1759 \r\n",
       "z\r\n",
       "M 3341 1997 \r\n",
       "L 3341 0 \r\n",
       "L 2766 0 \r\n",
       "L 2766 531 \r\n",
       "Q 2569 213 2275 61 \r\n",
       "Q 1981 -91 1556 -91 \r\n",
       "Q 1019 -91 701 211 \r\n",
       "Q 384 513 384 1019 \r\n",
       "Q 384 1609 779 1909 \r\n",
       "Q 1175 2209 1959 2209 \r\n",
       "L 2766 2209 \r\n",
       "L 2766 2266 \r\n",
       "Q 2766 2663 2505 2880 \r\n",
       "Q 2244 3097 1772 3097 \r\n",
       "Q 1472 3097 1187 3025 \r\n",
       "Q 903 2953 641 2809 \r\n",
       "L 641 3341 \r\n",
       "Q 956 3463 1253 3523 \r\n",
       "Q 1550 3584 1831 3584 \r\n",
       "Q 2591 3584 2966 3190 \r\n",
       "Q 3341 2797 3341 1997 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-61\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 603 3500 \r\n",
       "L 1178 3500 \r\n",
       "L 1178 0 \r\n",
       "L 603 0 \r\n",
       "L 603 3500 \r\n",
       "z\r\n",
       "M 603 4863 \r\n",
       "L 1178 4863 \r\n",
       "L 1178 4134 \r\n",
       "L 603 4134 \r\n",
       "L 603 4863 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-69\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 3513 2113 \r\n",
       "L 3513 0 \r\n",
       "L 2938 0 \r\n",
       "L 2938 2094 \r\n",
       "Q 2938 2591 2744 2837 \r\n",
       "Q 2550 3084 2163 3084 \r\n",
       "Q 1697 3084 1428 2787 \r\n",
       "Q 1159 2491 1159 1978 \r\n",
       "L 1159 0 \r\n",
       "L 581 0 \r\n",
       "L 581 3500 \r\n",
       "L 1159 3500 \r\n",
       "L 1159 2956 \r\n",
       "Q 1366 3272 1645 3428 \r\n",
       "Q 1925 3584 2291 3584 \r\n",
       "Q 2894 3584 3203 3211 \r\n",
       "Q 3513 2838 3513 2113 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-6e\" transform=\"scale(0.015625)\"/>\r\n",
       "      </defs>\r\n",
       "      <use xlink:href=\"#DejaVuSans-74\"/>\r\n",
       "      <use x=\"39.208984\" xlink:href=\"#DejaVuSans-72\"/>\r\n",
       "      <use x=\"80.322266\" xlink:href=\"#DejaVuSans-61\"/>\r\n",
       "      <use x=\"141.601562\" xlink:href=\"#DejaVuSans-69\"/>\r\n",
       "      <use x=\"169.384766\" xlink:href=\"#DejaVuSans-6e\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"line2d_23\">\r\n",
       "     <path d=\"M 60.378125 128.520312 \r\n",
       "L 80.378125 128.520312 \r\n",
       "\" style=\"fill:none;stroke:#bf00bf;stroke-dasharray:5.55,2.4;stroke-dashoffset:0;stroke-width:1.5;\"/>\r\n",
       "    </g>\r\n",
       "    <g id=\"line2d_24\"/>\r\n",
       "    <g id=\"text_13\">\r\n",
       "     <!-- test -->\r\n",
       "     <g transform=\"translate(88.378125 132.020312)scale(0.1 -0.1)\">\r\n",
       "      <use xlink:href=\"#DejaVuSans-74\"/>\r\n",
       "      <use x=\"39.208984\" xlink:href=\"#DejaVuSans-65\"/>\r\n",
       "      <use x=\"100.732422\" xlink:href=\"#DejaVuSans-73\"/>\r\n",
       "      <use x=\"152.832031\" xlink:href=\"#DejaVuSans-74\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "  </g>\r\n",
       " </g>\r\n",
       " <defs>\r\n",
       "  <clipPath id=\"pf89746b9ad\">\r\n",
       "   <rect height=\"135.9\" width=\"195.3\" x=\"51.378125\" y=\"7.2\"/>\r\n",
       "  </clipPath>\r\n",
       " </defs>\r\n",
       "</svg>\r\n"
      ],
      "text/plain": [
       "<Figure size 252x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "train_concise(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "origin_pos": 31,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "w的L2范数： 0.37420952320098877\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\r\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\r\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\r\n",
       "<svg height=\"180.65625pt\" version=\"1.1\" viewBox=\"0 0 263.421875 180.65625\" width=\"263.421875pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\r\n",
       " <metadata>\r\n",
       "  <rdf:RDF xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\r\n",
       "   <cc:Work>\r\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\r\n",
       "    <dc:date>2021-09-12T16:43:12.917569</dc:date>\r\n",
       "    <dc:format>image/svg+xml</dc:format>\r\n",
       "    <dc:creator>\r\n",
       "     <cc:Agent>\r\n",
       "      <dc:title>Matplotlib v3.4.3, https://matplotlib.org/</dc:title>\r\n",
       "     </cc:Agent>\r\n",
       "    </dc:creator>\r\n",
       "   </cc:Work>\r\n",
       "  </rdf:RDF>\r\n",
       " </metadata>\r\n",
       " <defs>\r\n",
       "  <style type=\"text/css\">*{stroke-linecap:butt;stroke-linejoin:round;}</style>\r\n",
       " </defs>\r\n",
       " <g id=\"figure_1\">\r\n",
       "  <g id=\"patch_1\">\r\n",
       "   <path d=\"M 0 180.65625 \r\n",
       "L 263.421875 180.65625 \r\n",
       "L 263.421875 0 \r\n",
       "L 0 0 \r\n",
       "z\r\n",
       "\" style=\"fill:none;\"/>\r\n",
       "  </g>\r\n",
       "  <g id=\"axes_1\">\r\n",
       "   <g id=\"patch_2\">\r\n",
       "    <path d=\"M 51.378125 143.1 \r\n",
       "L 246.678125 143.1 \r\n",
       "L 246.678125 7.2 \r\n",
       "L 51.378125 7.2 \r\n",
       "z\r\n",
       "\" style=\"fill:#ffffff;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"matplotlib.axis_1\">\r\n",
       "    <g id=\"xtick_1\">\r\n",
       "     <g id=\"line2d_1\">\r\n",
       "      <path clip-path=\"url(#p3e573c9733)\" d=\"M 82.214967 143.1 \r\n",
       "L 82.214967 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_2\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 0 0 \r\n",
       "L 0 3.5 \r\n",
       "\" id=\"mdc10a09464\" style=\"stroke:#000000;stroke-width:0.8;\"/>\r\n",
       "      </defs>\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"82.214967\" xlink:href=\"#mdc10a09464\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_1\">\r\n",
       "      <!-- 20 -->\r\n",
       "      <g transform=\"translate(75.852467 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 1228 531 \r\n",
       "L 3431 531 \r\n",
       "L 3431 0 \r\n",
       "L 469 0 \r\n",
       "L 469 531 \r\n",
       "Q 828 903 1448 1529 \r\n",
       "Q 2069 2156 2228 2338 \r\n",
       "Q 2531 2678 2651 2914 \r\n",
       "Q 2772 3150 2772 3378 \r\n",
       "Q 2772 3750 2511 3984 \r\n",
       "Q 2250 4219 1831 4219 \r\n",
       "Q 1534 4219 1204 4116 \r\n",
       "Q 875 4013 500 3803 \r\n",
       "L 500 4441 \r\n",
       "Q 881 4594 1212 4672 \r\n",
       "Q 1544 4750 1819 4750 \r\n",
       "Q 2544 4750 2975 4387 \r\n",
       "Q 3406 4025 3406 3419 \r\n",
       "Q 3406 3131 3298 2873 \r\n",
       "Q 3191 2616 2906 2266 \r\n",
       "Q 2828 2175 2409 1742 \r\n",
       "Q 1991 1309 1228 531 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-32\" transform=\"scale(0.015625)\"/>\r\n",
       "        <path d=\"M 2034 4250 \r\n",
       "Q 1547 4250 1301 3770 \r\n",
       "Q 1056 3291 1056 2328 \r\n",
       "Q 1056 1369 1301 889 \r\n",
       "Q 1547 409 2034 409 \r\n",
       "Q 2525 409 2770 889 \r\n",
       "Q 3016 1369 3016 2328 \r\n",
       "Q 3016 3291 2770 3770 \r\n",
       "Q 2525 4250 2034 4250 \r\n",
       "z\r\n",
       "M 2034 4750 \r\n",
       "Q 2819 4750 3233 4129 \r\n",
       "Q 3647 3509 3647 2328 \r\n",
       "Q 3647 1150 3233 529 \r\n",
       "Q 2819 -91 2034 -91 \r\n",
       "Q 1250 -91 836 529 \r\n",
       "Q 422 1150 422 2328 \r\n",
       "Q 422 3509 836 4129 \r\n",
       "Q 1250 4750 2034 4750 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-30\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_2\">\r\n",
       "     <g id=\"line2d_3\">\r\n",
       "      <path clip-path=\"url(#p3e573c9733)\" d=\"M 123.330757 143.1 \r\n",
       "L 123.330757 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_4\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"123.330757\" xlink:href=\"#mdc10a09464\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_2\">\r\n",
       "      <!-- 40 -->\r\n",
       "      <g transform=\"translate(116.968257 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 2419 4116 \r\n",
       "L 825 1625 \r\n",
       "L 2419 1625 \r\n",
       "L 2419 4116 \r\n",
       "z\r\n",
       "M 2253 4666 \r\n",
       "L 3047 4666 \r\n",
       "L 3047 1625 \r\n",
       "L 3713 1625 \r\n",
       "L 3713 1100 \r\n",
       "L 3047 1100 \r\n",
       "L 3047 0 \r\n",
       "L 2419 0 \r\n",
       "L 2419 1100 \r\n",
       "L 313 1100 \r\n",
       "L 313 1709 \r\n",
       "L 2253 4666 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-34\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-34\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_3\">\r\n",
       "     <g id=\"line2d_5\">\r\n",
       "      <path clip-path=\"url(#p3e573c9733)\" d=\"M 164.446546 143.1 \r\n",
       "L 164.446546 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_6\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"164.446546\" xlink:href=\"#mdc10a09464\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_3\">\r\n",
       "      <!-- 60 -->\r\n",
       "      <g transform=\"translate(158.084046 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 2113 2584 \r\n",
       "Q 1688 2584 1439 2293 \r\n",
       "Q 1191 2003 1191 1497 \r\n",
       "Q 1191 994 1439 701 \r\n",
       "Q 1688 409 2113 409 \r\n",
       "Q 2538 409 2786 701 \r\n",
       "Q 3034 994 3034 1497 \r\n",
       "Q 3034 2003 2786 2293 \r\n",
       "Q 2538 2584 2113 2584 \r\n",
       "z\r\n",
       "M 3366 4563 \r\n",
       "L 3366 3988 \r\n",
       "Q 3128 4100 2886 4159 \r\n",
       "Q 2644 4219 2406 4219 \r\n",
       "Q 1781 4219 1451 3797 \r\n",
       "Q 1122 3375 1075 2522 \r\n",
       "Q 1259 2794 1537 2939 \r\n",
       "Q 1816 3084 2150 3084 \r\n",
       "Q 2853 3084 3261 2657 \r\n",
       "Q 3669 2231 3669 1497 \r\n",
       "Q 3669 778 3244 343 \r\n",
       "Q 2819 -91 2113 -91 \r\n",
       "Q 1303 -91 875 529 \r\n",
       "Q 447 1150 447 2328 \r\n",
       "Q 447 3434 972 4092 \r\n",
       "Q 1497 4750 2381 4750 \r\n",
       "Q 2619 4750 2861 4703 \r\n",
       "Q 3103 4656 3366 4563 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-36\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-36\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_4\">\r\n",
       "     <g id=\"line2d_7\">\r\n",
       "      <path clip-path=\"url(#p3e573c9733)\" d=\"M 205.562336 143.1 \r\n",
       "L 205.562336 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_8\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"205.562336\" xlink:href=\"#mdc10a09464\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_4\">\r\n",
       "      <!-- 80 -->\r\n",
       "      <g transform=\"translate(199.199836 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 2034 2216 \r\n",
       "Q 1584 2216 1326 1975 \r\n",
       "Q 1069 1734 1069 1313 \r\n",
       "Q 1069 891 1326 650 \r\n",
       "Q 1584 409 2034 409 \r\n",
       "Q 2484 409 2743 651 \r\n",
       "Q 3003 894 3003 1313 \r\n",
       "Q 3003 1734 2745 1975 \r\n",
       "Q 2488 2216 2034 2216 \r\n",
       "z\r\n",
       "M 1403 2484 \r\n",
       "Q 997 2584 770 2862 \r\n",
       "Q 544 3141 544 3541 \r\n",
       "Q 544 4100 942 4425 \r\n",
       "Q 1341 4750 2034 4750 \r\n",
       "Q 2731 4750 3128 4425 \r\n",
       "Q 3525 4100 3525 3541 \r\n",
       "Q 3525 3141 3298 2862 \r\n",
       "Q 3072 2584 2669 2484 \r\n",
       "Q 3125 2378 3379 2068 \r\n",
       "Q 3634 1759 3634 1313 \r\n",
       "Q 3634 634 3220 271 \r\n",
       "Q 2806 -91 2034 -91 \r\n",
       "Q 1263 -91 848 271 \r\n",
       "Q 434 634 434 1313 \r\n",
       "Q 434 1759 690 2068 \r\n",
       "Q 947 2378 1403 2484 \r\n",
       "z\r\n",
       "M 1172 3481 \r\n",
       "Q 1172 3119 1398 2916 \r\n",
       "Q 1625 2713 2034 2713 \r\n",
       "Q 2441 2713 2670 2916 \r\n",
       "Q 2900 3119 2900 3481 \r\n",
       "Q 2900 3844 2670 4047 \r\n",
       "Q 2441 4250 2034 4250 \r\n",
       "Q 1625 4250 1398 4047 \r\n",
       "Q 1172 3844 1172 3481 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-38\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-38\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_5\">\r\n",
       "     <g id=\"line2d_9\">\r\n",
       "      <path clip-path=\"url(#p3e573c9733)\" d=\"M 246.678125 143.1 \r\n",
       "L 246.678125 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_10\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"246.678125\" xlink:href=\"#mdc10a09464\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_5\">\r\n",
       "      <!-- 100 -->\r\n",
       "      <g transform=\"translate(237.134375 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 794 531 \r\n",
       "L 1825 531 \r\n",
       "L 1825 4091 \r\n",
       "L 703 3866 \r\n",
       "L 703 4441 \r\n",
       "L 1819 4666 \r\n",
       "L 2450 4666 \r\n",
       "L 2450 531 \r\n",
       "L 3481 531 \r\n",
       "L 3481 0 \r\n",
       "L 794 0 \r\n",
       "L 794 531 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-31\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"text_6\">\r\n",
       "     <!-- epochs -->\r\n",
       "     <g transform=\"translate(131.195312 171.376563)scale(0.1 -0.1)\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 3597 1894 \r\n",
       "L 3597 1613 \r\n",
       "L 953 1613 \r\n",
       "Q 991 1019 1311 708 \r\n",
       "Q 1631 397 2203 397 \r\n",
       "Q 2534 397 2845 478 \r\n",
       "Q 3156 559 3463 722 \r\n",
       "L 3463 178 \r\n",
       "Q 3153 47 2828 -22 \r\n",
       "Q 2503 -91 2169 -91 \r\n",
       "Q 1331 -91 842 396 \r\n",
       "Q 353 884 353 1716 \r\n",
       "Q 353 2575 817 3079 \r\n",
       "Q 1281 3584 2069 3584 \r\n",
       "Q 2775 3584 3186 3129 \r\n",
       "Q 3597 2675 3597 1894 \r\n",
       "z\r\n",
       "M 3022 2063 \r\n",
       "Q 3016 2534 2758 2815 \r\n",
       "Q 2500 3097 2075 3097 \r\n",
       "Q 1594 3097 1305 2825 \r\n",
       "Q 1016 2553 972 2059 \r\n",
       "L 3022 2063 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-65\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 1159 525 \r\n",
       "L 1159 -1331 \r\n",
       "L 581 -1331 \r\n",
       "L 581 3500 \r\n",
       "L 1159 3500 \r\n",
       "L 1159 2969 \r\n",
       "Q 1341 3281 1617 3432 \r\n",
       "Q 1894 3584 2278 3584 \r\n",
       "Q 2916 3584 3314 3078 \r\n",
       "Q 3713 2572 3713 1747 \r\n",
       "Q 3713 922 3314 415 \r\n",
       "Q 2916 -91 2278 -91 \r\n",
       "Q 1894 -91 1617 61 \r\n",
       "Q 1341 213 1159 525 \r\n",
       "z\r\n",
       "M 3116 1747 \r\n",
       "Q 3116 2381 2855 2742 \r\n",
       "Q 2594 3103 2138 3103 \r\n",
       "Q 1681 3103 1420 2742 \r\n",
       "Q 1159 2381 1159 1747 \r\n",
       "Q 1159 1113 1420 752 \r\n",
       "Q 1681 391 2138 391 \r\n",
       "Q 2594 391 2855 752 \r\n",
       "Q 3116 1113 3116 1747 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-70\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 1959 3097 \r\n",
       "Q 1497 3097 1228 2736 \r\n",
       "Q 959 2375 959 1747 \r\n",
       "Q 959 1119 1226 758 \r\n",
       "Q 1494 397 1959 397 \r\n",
       "Q 2419 397 2687 759 \r\n",
       "Q 2956 1122 2956 1747 \r\n",
       "Q 2956 2369 2687 2733 \r\n",
       "Q 2419 3097 1959 3097 \r\n",
       "z\r\n",
       "M 1959 3584 \r\n",
       "Q 2709 3584 3137 3096 \r\n",
       "Q 3566 2609 3566 1747 \r\n",
       "Q 3566 888 3137 398 \r\n",
       "Q 2709 -91 1959 -91 \r\n",
       "Q 1206 -91 779 398 \r\n",
       "Q 353 888 353 1747 \r\n",
       "Q 353 2609 779 3096 \r\n",
       "Q 1206 3584 1959 3584 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-6f\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 3122 3366 \r\n",
       "L 3122 2828 \r\n",
       "Q 2878 2963 2633 3030 \r\n",
       "Q 2388 3097 2138 3097 \r\n",
       "Q 1578 3097 1268 2742 \r\n",
       "Q 959 2388 959 1747 \r\n",
       "Q 959 1106 1268 751 \r\n",
       "Q 1578 397 2138 397 \r\n",
       "Q 2388 397 2633 464 \r\n",
       "Q 2878 531 3122 666 \r\n",
       "L 3122 134 \r\n",
       "Q 2881 22 2623 -34 \r\n",
       "Q 2366 -91 2075 -91 \r\n",
       "Q 1284 -91 818 406 \r\n",
       "Q 353 903 353 1747 \r\n",
       "Q 353 2603 823 3093 \r\n",
       "Q 1294 3584 2113 3584 \r\n",
       "Q 2378 3584 2631 3529 \r\n",
       "Q 2884 3475 3122 3366 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-63\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 3513 2113 \r\n",
       "L 3513 0 \r\n",
       "L 2938 0 \r\n",
       "L 2938 2094 \r\n",
       "Q 2938 2591 2744 2837 \r\n",
       "Q 2550 3084 2163 3084 \r\n",
       "Q 1697 3084 1428 2787 \r\n",
       "Q 1159 2491 1159 1978 \r\n",
       "L 1159 0 \r\n",
       "L 581 0 \r\n",
       "L 581 4863 \r\n",
       "L 1159 4863 \r\n",
       "L 1159 2956 \r\n",
       "Q 1366 3272 1645 3428 \r\n",
       "Q 1925 3584 2291 3584 \r\n",
       "Q 2894 3584 3203 3211 \r\n",
       "Q 3513 2838 3513 2113 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-68\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 2834 3397 \r\n",
       "L 2834 2853 \r\n",
       "Q 2591 2978 2328 3040 \r\n",
       "Q 2066 3103 1784 3103 \r\n",
       "Q 1356 3103 1142 2972 \r\n",
       "Q 928 2841 928 2578 \r\n",
       "Q 928 2378 1081 2264 \r\n",
       "Q 1234 2150 1697 2047 \r\n",
       "L 1894 2003 \r\n",
       "Q 2506 1872 2764 1633 \r\n",
       "Q 3022 1394 3022 966 \r\n",
       "Q 3022 478 2636 193 \r\n",
       "Q 2250 -91 1575 -91 \r\n",
       "Q 1294 -91 989 -36 \r\n",
       "Q 684 19 347 128 \r\n",
       "L 347 722 \r\n",
       "Q 666 556 975 473 \r\n",
       "Q 1284 391 1588 391 \r\n",
       "Q 1994 391 2212 530 \r\n",
       "Q 2431 669 2431 922 \r\n",
       "Q 2431 1156 2273 1281 \r\n",
       "Q 2116 1406 1581 1522 \r\n",
       "L 1381 1569 \r\n",
       "Q 847 1681 609 1914 \r\n",
       "Q 372 2147 372 2553 \r\n",
       "Q 372 3047 722 3315 \r\n",
       "Q 1072 3584 1716 3584 \r\n",
       "Q 2034 3584 2315 3537 \r\n",
       "Q 2597 3491 2834 3397 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-73\" transform=\"scale(0.015625)\"/>\r\n",
       "      </defs>\r\n",
       "      <use xlink:href=\"#DejaVuSans-65\"/>\r\n",
       "      <use x=\"61.523438\" xlink:href=\"#DejaVuSans-70\"/>\r\n",
       "      <use x=\"125\" xlink:href=\"#DejaVuSans-6f\"/>\r\n",
       "      <use x=\"186.181641\" xlink:href=\"#DejaVuSans-63\"/>\r\n",
       "      <use x=\"241.162109\" xlink:href=\"#DejaVuSans-68\"/>\r\n",
       "      <use x=\"304.541016\" xlink:href=\"#DejaVuSans-73\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "   <g id=\"matplotlib.axis_2\">\r\n",
       "    <g id=\"ytick_1\">\r\n",
       "     <g id=\"line2d_11\">\r\n",
       "      <path clip-path=\"url(#p3e573c9733)\" d=\"M 51.378125 104.230808 \r\n",
       "L 246.678125 104.230808 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_12\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 0 0 \r\n",
       "L -3.5 0 \r\n",
       "\" id=\"m3a19726d59\" style=\"stroke:#000000;stroke-width:0.8;\"/>\r\n",
       "      </defs>\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"51.378125\" xlink:href=\"#m3a19726d59\" y=\"104.230808\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_7\">\r\n",
       "      <!-- $\\mathdefault{10^{-2}}$ -->\r\n",
       "      <g transform=\"translate(20.878125 108.030027)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 678 2272 \r\n",
       "L 4684 2272 \r\n",
       "L 4684 1741 \r\n",
       "L 678 1741 \r\n",
       "L 678 2272 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-2212\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use transform=\"translate(0 0.765625)\" xlink:href=\"#DejaVuSans-31\"/>\r\n",
       "       <use transform=\"translate(63.623047 0.765625)\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use transform=\"translate(128.203125 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-2212\"/>\r\n",
       "       <use transform=\"translate(186.855469 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-32\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_2\">\r\n",
       "     <g id=\"line2d_13\">\r\n",
       "      <path clip-path=\"url(#p3e573c9733)\" d=\"M 51.378125 60.135454 \r\n",
       "L 246.678125 60.135454 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_14\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"51.378125\" xlink:href=\"#m3a19726d59\" y=\"60.135454\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_8\">\r\n",
       "      <!-- $\\mathdefault{10^{0}}$ -->\r\n",
       "      <g transform=\"translate(26.778125 63.934673)scale(0.1 -0.1)\">\r\n",
       "       <use transform=\"translate(0 0.765625)\" xlink:href=\"#DejaVuSans-31\"/>\r\n",
       "       <use transform=\"translate(63.623047 0.765625)\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use transform=\"translate(128.203125 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_3\">\r\n",
       "     <g id=\"line2d_15\">\r\n",
       "      <path clip-path=\"url(#p3e573c9733)\" d=\"M 51.378125 16.0401 \r\n",
       "L 246.678125 16.0401 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_16\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"51.378125\" xlink:href=\"#m3a19726d59\" y=\"16.0401\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_9\">\r\n",
       "      <!-- $\\mathdefault{10^{2}}$ -->\r\n",
       "      <g transform=\"translate(26.778125 19.839319)scale(0.1 -0.1)\">\r\n",
       "       <use transform=\"translate(0 0.765625)\" xlink:href=\"#DejaVuSans-31\"/>\r\n",
       "       <use transform=\"translate(63.623047 0.765625)\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use transform=\"translate(128.203125 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-32\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"text_10\">\r\n",
       "     <!-- loss -->\r\n",
       "     <g transform=\"translate(14.798437 84.807812)rotate(-90)scale(0.1 -0.1)\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 603 4863 \r\n",
       "L 1178 4863 \r\n",
       "L 1178 0 \r\n",
       "L 603 0 \r\n",
       "L 603 4863 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-6c\" transform=\"scale(0.015625)\"/>\r\n",
       "      </defs>\r\n",
       "      <use xlink:href=\"#DejaVuSans-6c\"/>\r\n",
       "      <use x=\"27.783203\" xlink:href=\"#DejaVuSans-6f\"/>\r\n",
       "      <use x=\"88.964844\" xlink:href=\"#DejaVuSans-73\"/>\r\n",
       "      <use x=\"141.064453\" xlink:href=\"#DejaVuSans-73\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "   <g id=\"line2d_17\">\r\n",
       "    <path clip-path=\"url(#p3e573c9733)\" d=\"M 51.378125 33.888055 \r\n",
       "L 61.657072 55.91405 \r\n",
       "L 71.93602 76.721544 \r\n",
       "L 82.214967 97.224755 \r\n",
       "L 92.493914 117.444084 \r\n",
       "L 102.772862 133.148471 \r\n",
       "L 113.051809 136.922727 \r\n",
       "L 123.330757 136.159976 \r\n",
       "L 133.609704 135.996933 \r\n",
       "L 143.888651 135.442877 \r\n",
       "L 154.167599 135.578079 \r\n",
       "L 164.446546 135.319022 \r\n",
       "L 174.725493 135.52448 \r\n",
       "L 185.004441 135.195116 \r\n",
       "L 195.283388 135.195451 \r\n",
       "L 205.562336 135.508249 \r\n",
       "L 215.841283 135.487191 \r\n",
       "L 226.12023 135.515312 \r\n",
       "L 236.399178 135.737168 \r\n",
       "L 246.678125 135.770455 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-linecap:square;stroke-width:1.5;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"line2d_18\">\r\n",
       "    <path clip-path=\"url(#p3e573c9733)\" d=\"M 51.378125 13.377273 \r\n",
       "L 61.657072 17.084613 \r\n",
       "L 71.93602 20.604496 \r\n",
       "L 82.214967 24.078298 \r\n",
       "L 92.493914 27.536561 \r\n",
       "L 102.772862 30.986466 \r\n",
       "L 113.051809 34.430581 \r\n",
       "L 123.330757 37.86906 \r\n",
       "L 133.609704 41.30137 \r\n",
       "L 143.888651 44.726214 \r\n",
       "L 154.167599 48.141356 \r\n",
       "L 164.446546 51.544692 \r\n",
       "L 174.725493 54.932415 \r\n",
       "L 185.004441 58.299861 \r\n",
       "L 195.283388 61.641238 \r\n",
       "L 205.562336 64.947889 \r\n",
       "L 215.841283 68.210382 \r\n",
       "L 226.12023 71.415096 \r\n",
       "L 236.399178 74.544889 \r\n",
       "L 246.678125 77.577184 \r\n",
       "\" style=\"fill:none;stroke:#bf00bf;stroke-dasharray:5.55,2.4;stroke-dashoffset:0;stroke-width:1.5;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_3\">\r\n",
       "    <path d=\"M 51.378125 143.1 \r\n",
       "L 51.378125 7.2 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_4\">\r\n",
       "    <path d=\"M 246.678125 143.1 \r\n",
       "L 246.678125 7.2 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_5\">\r\n",
       "    <path d=\"M 51.378125 143.1 \r\n",
       "L 246.678125 143.1 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_6\">\r\n",
       "    <path d=\"M 51.378125 7.2 \r\n",
       "L 246.678125 7.2 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"legend_1\">\r\n",
       "    <g id=\"patch_7\">\r\n",
       "     <path d=\"M 184.403125 44.55625 \r\n",
       "L 239.678125 44.55625 \r\n",
       "Q 241.678125 44.55625 241.678125 42.55625 \r\n",
       "L 241.678125 14.2 \r\n",
       "Q 241.678125 12.2 239.678125 12.2 \r\n",
       "L 184.403125 12.2 \r\n",
       "Q 182.403125 12.2 182.403125 14.2 \r\n",
       "L 182.403125 42.55625 \r\n",
       "Q 182.403125 44.55625 184.403125 44.55625 \r\n",
       "z\r\n",
       "\" style=\"fill:#ffffff;opacity:0.8;stroke:#cccccc;stroke-linejoin:miter;\"/>\r\n",
       "    </g>\r\n",
       "    <g id=\"line2d_19\">\r\n",
       "     <path d=\"M 186.403125 20.298437 \r\n",
       "L 206.403125 20.298437 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-linecap:square;stroke-width:1.5;\"/>\r\n",
       "    </g>\r\n",
       "    <g id=\"line2d_20\"/>\r\n",
       "    <g id=\"text_11\">\r\n",
       "     <!-- train -->\r\n",
       "     <g transform=\"translate(214.403125 23.798437)scale(0.1 -0.1)\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 1172 4494 \r\n",
       "L 1172 3500 \r\n",
       "L 2356 3500 \r\n",
       "L 2356 3053 \r\n",
       "L 1172 3053 \r\n",
       "L 1172 1153 \r\n",
       "Q 1172 725 1289 603 \r\n",
       "Q 1406 481 1766 481 \r\n",
       "L 2356 481 \r\n",
       "L 2356 0 \r\n",
       "L 1766 0 \r\n",
       "Q 1100 0 847 248 \r\n",
       "Q 594 497 594 1153 \r\n",
       "L 594 3053 \r\n",
       "L 172 3053 \r\n",
       "L 172 3500 \r\n",
       "L 594 3500 \r\n",
       "L 594 4494 \r\n",
       "L 1172 4494 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-74\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 2631 2963 \r\n",
       "Q 2534 3019 2420 3045 \r\n",
       "Q 2306 3072 2169 3072 \r\n",
       "Q 1681 3072 1420 2755 \r\n",
       "Q 1159 2438 1159 1844 \r\n",
       "L 1159 0 \r\n",
       "L 581 0 \r\n",
       "L 581 3500 \r\n",
       "L 1159 3500 \r\n",
       "L 1159 2956 \r\n",
       "Q 1341 3275 1631 3429 \r\n",
       "Q 1922 3584 2338 3584 \r\n",
       "Q 2397 3584 2469 3576 \r\n",
       "Q 2541 3569 2628 3553 \r\n",
       "L 2631 2963 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-72\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 2194 1759 \r\n",
       "Q 1497 1759 1228 1600 \r\n",
       "Q 959 1441 959 1056 \r\n",
       "Q 959 750 1161 570 \r\n",
       "Q 1363 391 1709 391 \r\n",
       "Q 2188 391 2477 730 \r\n",
       "Q 2766 1069 2766 1631 \r\n",
       "L 2766 1759 \r\n",
       "L 2194 1759 \r\n",
       "z\r\n",
       "M 3341 1997 \r\n",
       "L 3341 0 \r\n",
       "L 2766 0 \r\n",
       "L 2766 531 \r\n",
       "Q 2569 213 2275 61 \r\n",
       "Q 1981 -91 1556 -91 \r\n",
       "Q 1019 -91 701 211 \r\n",
       "Q 384 513 384 1019 \r\n",
       "Q 384 1609 779 1909 \r\n",
       "Q 1175 2209 1959 2209 \r\n",
       "L 2766 2209 \r\n",
       "L 2766 2266 \r\n",
       "Q 2766 2663 2505 2880 \r\n",
       "Q 2244 3097 1772 3097 \r\n",
       "Q 1472 3097 1187 3025 \r\n",
       "Q 903 2953 641 2809 \r\n",
       "L 641 3341 \r\n",
       "Q 956 3463 1253 3523 \r\n",
       "Q 1550 3584 1831 3584 \r\n",
       "Q 2591 3584 2966 3190 \r\n",
       "Q 3341 2797 3341 1997 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-61\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 603 3500 \r\n",
       "L 1178 3500 \r\n",
       "L 1178 0 \r\n",
       "L 603 0 \r\n",
       "L 603 3500 \r\n",
       "z\r\n",
       "M 603 4863 \r\n",
       "L 1178 4863 \r\n",
       "L 1178 4134 \r\n",
       "L 603 4134 \r\n",
       "L 603 4863 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-69\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 3513 2113 \r\n",
       "L 3513 0 \r\n",
       "L 2938 0 \r\n",
       "L 2938 2094 \r\n",
       "Q 2938 2591 2744 2837 \r\n",
       "Q 2550 3084 2163 3084 \r\n",
       "Q 1697 3084 1428 2787 \r\n",
       "Q 1159 2491 1159 1978 \r\n",
       "L 1159 0 \r\n",
       "L 581 0 \r\n",
       "L 581 3500 \r\n",
       "L 1159 3500 \r\n",
       "L 1159 2956 \r\n",
       "Q 1366 3272 1645 3428 \r\n",
       "Q 1925 3584 2291 3584 \r\n",
       "Q 2894 3584 3203 3211 \r\n",
       "Q 3513 2838 3513 2113 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-6e\" transform=\"scale(0.015625)\"/>\r\n",
       "      </defs>\r\n",
       "      <use xlink:href=\"#DejaVuSans-74\"/>\r\n",
       "      <use x=\"39.208984\" xlink:href=\"#DejaVuSans-72\"/>\r\n",
       "      <use x=\"80.322266\" xlink:href=\"#DejaVuSans-61\"/>\r\n",
       "      <use x=\"141.601562\" xlink:href=\"#DejaVuSans-69\"/>\r\n",
       "      <use x=\"169.384766\" xlink:href=\"#DejaVuSans-6e\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"line2d_21\">\r\n",
       "     <path d=\"M 186.403125 34.976562 \r\n",
       "L 206.403125 34.976562 \r\n",
       "\" style=\"fill:none;stroke:#bf00bf;stroke-dasharray:5.55,2.4;stroke-dashoffset:0;stroke-width:1.5;\"/>\r\n",
       "    </g>\r\n",
       "    <g id=\"line2d_22\"/>\r\n",
       "    <g id=\"text_12\">\r\n",
       "     <!-- test -->\r\n",
       "     <g transform=\"translate(214.403125 38.476562)scale(0.1 -0.1)\">\r\n",
       "      <use xlink:href=\"#DejaVuSans-74\"/>\r\n",
       "      <use x=\"39.208984\" xlink:href=\"#DejaVuSans-65\"/>\r\n",
       "      <use x=\"100.732422\" xlink:href=\"#DejaVuSans-73\"/>\r\n",
       "      <use x=\"152.832031\" xlink:href=\"#DejaVuSans-74\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "  </g>\r\n",
       " </g>\r\n",
       " <defs>\r\n",
       "  <clipPath id=\"p3e573c9733\">\r\n",
       "   <rect height=\"135.9\" width=\"195.3\" x=\"51.378125\" y=\"7.2\"/>\r\n",
       "  </clipPath>\r\n",
       " </defs>\r\n",
       "</svg>\r\n"
      ],
      "text/plain": [
       "<Figure size 252x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "train_concise(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 32
   },
   "source": [
    "到目前为止，我们只接触到一个简单线性函数的概念。此外，由什么构成一个简单的非线性函数可能是一个更复杂的问题。例如，[再生核希尔伯特空间（RKHS）](https://en.wikipedia.org/wiki/Reproducing_kernel_Hilbert_space)允许在非线性环境中应用为线性函数引入的工具。不幸的是，基于RKHS的算法往往难以扩展到大型、高维的数据。在这本书中，我们将默认使用简单的启发式方法，即在深层网络的所有层上应用权重衰减。\n",
    "\n",
    "## 小结\n",
    "\n",
    "* 正则化是处理过拟合的常用方法。在训练集的损失函数中加入惩罚项，以降低学习到的模型的复杂度。\n",
    "* 保持模型简单的一个特别的选择是使用$L_2$惩罚的权重衰减。这会导致学习算法更新步骤中的权重衰减。\n",
    "* 权重衰减功能在深度学习框架的优化器中提供。\n",
    "* 在同一训练代码实现中，不同的参数集可以有不同的更新行为。\n",
    "\n",
    "## 练习\n",
    "\n",
    "1. 在本节的估计问题中使用$\\lambda$的值进行实验。绘制训练和测试准确率关于$\\lambda$的函数。你观察到了什么？\n",
    "1. 使用验证集来找到最佳值$\\lambda$。它真的是最优值吗？这有关系吗？\n",
    "1. 如果我们使用$\\sum_i |w_i|$作为我们选择的惩罚（$L_1$正则化），那么更新方程会是什么样子？\n",
    "1. 我们知道$\\|\\mathbf{w}\\|^2 = \\mathbf{w}^\\top \\mathbf{w}$。你能找到类似的矩阵方程吗（见 :numref:`subsec_lin-algebra-norms` 中的弗罗贝尼乌斯范数）？\n",
    "1. 回顾训练误差和泛化误差之间的关系。除了权重衰减、增加训练数据、使用适当复杂度的模型之外，你还能想出其他什么方法来处理过拟合？\n",
    "1. 在贝叶斯统计中，我们使用先验和似然的乘积，通过公式$P(w \\mid x) \\propto P(x \\mid w) P(w)$得到后验。如何得到带正则化的$P(w)$？\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 34,
    "tab": [
     "pytorch"
    ]
   },
   "source": [
    "[Discussions](https://discuss.d2l.ai/t/1808)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "pte",
   "language": "python",
   "name": "pte"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
